| name | mixture-of-experts-adaptive-capacity |
| title | DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for MoE |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.01697 |
| keywords | ["Mixture of Experts","Adaptive Routing","Token-Level Dynamics","Capacity Planning","Sparse Models"] |
| description | Optimize Mixture-of-Experts efficiency by decoupling token-level expert activation from layer architecture. Use dynamic threshold routing where expert count per token varies by input complexity, and apply layer-wise capacity scheduling to match representational diversity patterns. |
DynaMoE: Adaptive Expert Capacity Allocation in Mixture-of-Experts
Standard Mixture-of-Experts (MoE) architectures use fixed Top-K routing, where exactly K experts activate per token across all layers. This constraint wastes capacity on simple inputs and under-scales complex ones. DynaMoE relaxes both constraints: enabling dynamic per-token expert counts and layer-dependent expert capacity scheduling to match the representational structure of deep networks.
The core insight is that different layers have different optimal expert counts based on input diversity. Early layers process heterogeneous raw inputs requiring diverse expert specialization; deeper layers exhibit representational convergence where fewer experts suffice. By jointly optimizing token routing flexibility and layer capacity distribution, DynaMoE achieves superior efficiency-performance trade-offs.
Core Concept
DynaMoE makes two related innovations:
-
Token-Level Threshold Routing: Replace fixed Top-K with a percentile threshold where expert count varies per token based on input-specific routing logits. Simple inputs use 1-2 experts; complex inputs use more.
-
Layer-Wise Capacity Scheduling: Distribute total expert budget across layers using scheduling strategies (descending, ascending, pyramid, wave) to match how representational complexity evolves through the network.
This two-level decomposition aligns model capacity with actual computational needs rather than fixed architectural assumptions.
Architecture Overview
- Input: Token sequences with per-token routing scores from gating network
- Dynamic Threshold Extraction: For each token, compute expert activation threshold based on percentile of routing logits
- Expert Selection: Activate all experts exceeding the threshold (variable count per token)
- Capacity Budget Distribution: Assign expert counts to layers via scheduling strategy
- Output: Efficient sparse forward/backward passes with adaptive activation patterns
Implementation Steps
Step 1: Compute per-token routing logits and dynamic thresholds
Standard MoE gating networks output logits for each expert. Instead of selecting top-K, use percentile-based thresholding to determine active experts adaptively.
batch_size, seq_len, num_experts = tokens.shape[0], tokens.shape[1], 128
logits = gating_network(tokens)
percentile =
thresholds = np.percentile(logits, percentile, axis=-, keepdims=)
active_mask = (logits >= thresholds).astype(np.float32)
num_active = active_mask.(axis=-)
