moe-routing-alignment

name	moe-routing-alignment
title	Stabilizing MoE RL by Aligning Training and Inference Routers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.11370
keywords	["MoE","Routing","Reinforcement Learning","Stability","Training"]
description	Prevents MoE router instability during RL training by recording and replaying inference-phase routing distributions back into training. Reduces training-inference routing divergence and KL divergence, enabling stable MoE RL scaling without sacrificing training speed.

Rollout Routing Replay: Stabilizing Mixture-of-Experts RL

Mixture-of-Experts models with RL training suffer from routing instability: routers diverge between training and inference phases, causing catastrophic collapse. Rollout Routing Replay (R3) synchronizes routing behavior across both phases.

By recording inference routing and replaying those decisions during training, R3 prevents divergence and enables stable scaling.

Core Concept

The core problem: routers make different decisions during training vs. inference, even with identical inputs. This divergence causes:

• Policy collapse in RL training
• Inconsistent expert utilization
• Degraded final model performance

R3 solves this by:

• Recording routing decisions during inference rollouts
• Replaying those same routing distributions during policy gradient training
• Synchronizing training and inference for the same states

Architecture Overview

• Inference-phase routing distribution recording
• Training-phase replay mechanism to enforce learned routing
• KL divergence minimization between phases
• Efficient storage of routing patterns

Implementation Steps

Record routing distributions during inference rollouts. When the model makes decisions in the environment, capture which experts were selected:

class RoutingRecorder:
    def __init__(self, num_experts):
        self.num_experts = num_experts
        self.routing_history = []

    def record_routing(self, inputs, routing_logits):
        """Capture routing decisions during inference."""
        # Get expert selection distribution
        routing_probs = torch.softmax(routing_logits, dim=-1)

        # For each token, record which expert was selected
        selected_experts = torch.argmax(routing_probs, dim=-1)

        record = {
            'inputs': inputs.detach().cpu(),
            'routing_logits': routing_logits.detach().cpu(),
            : routing_probs.detach().cpu(),
            : selected_experts.detach().cpu()
        }

        .routing_history.append(record)
         record

     ():
        
         .routing_history

     ():
        
        .routing_history = []