菜单
Optimize Mixture-of-Experts efficiency by decoupling token-level expert activation from layer architecture. Use dynamic threshold routing where expert count per token varies by input complexity, and apply layer-wise capacity scheduling to match representational diversity patterns.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill mixture-of-experts-adaptive-capacity命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
基于 SOC 职业分类
| name | mixture-of-experts-adaptive-capacity |
| title | DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for MoE |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.01697 |
| keywords | ["Mixture of Experts","Adaptive Routing","Token-Level Dynamics","Capacity Planning","Sparse Models"] |
| description | Optimize Mixture-of-Experts efficiency by decoupling token-level expert activation from layer architecture. Use dynamic threshold routing where expert count per token varies by input complexity, and apply layer-wise capacity scheduling to match representational diversity patterns. |
Standard Mixture-of-Experts (MoE) architectures use fixed Top-K routing, where exactly K experts activate per token across all layers. This constraint wastes capacity on simple inputs and under-scales complex ones. DynaMoE relaxes both constraints: enabling dynamic per-token expert counts and layer-dependent expert capacity scheduling to match the representational structure of deep networks.
The core insight is that different layers have different optimal expert counts based on input diversity. Early layers process heterogeneous raw inputs requiring diverse expert specialization; deeper layers exhibit representational convergence where fewer experts suffice. By jointly optimizing token routing flexibility and layer capacity distribution, DynaMoE achieves superior efficiency-performance trade-offs.
DynaMoE makes two related innovations:
Token-Level Threshold Routing: Replace fixed Top-K with a percentile threshold where expert count varies per token based on input-specific routing logits. Simple inputs use 1-2 experts; complex inputs use more.
Layer-Wise Capacity Scheduling: Distribute total expert budget across layers using scheduling strategies (descending, ascending, pyramid, wave) to match how representational complexity evolves through the network.
This two-level decomposition aligns model capacity with actual computational needs rather than fixed architectural assumptions.
Step 1: Compute per-token routing logits and dynamic thresholds
Standard MoE gating networks output logits for each expert. Instead of selecting top-K, use percentile-based thresholding to determine active experts adaptively.
# Forward pass through MoE gating network
batch_size, seq_len, num_experts = tokens.shape[0], tokens.shape[1], 128
logits = gating_network(tokens)
percentile =
thresholds = np.percentile(logits, percentile, axis=-, keepdims=)
active_mask = (logits >= thresholds).astype(np.float32)
num_active = active_mask.(axis=-)
Step 2: Implement capacity scheduling strategy
Distribute expert budget across layers based on representational diversity patterns. Choose scheduling strategy aligned with task structure.
def layer_capacity_schedule(layer_idx, num_layers, total_experts, strategy='descending'):
"""Allocate expert count per layer."""
if strategy == 'descending':
# More experts early (diverse raw input), fewer deep (converged representations)
# Quadratic decay for smooth transition
ratio = (num_layers - layer_idx) / num_layers
return int(total_experts * (ratio ** 2))
elif strategy == 'ascending':
# Fewer experts early, more deep
ratio = layer_idx / num_layers
return int(total_experts * (ratio ** 2))
elif strategy == 'pyramid':
# Peak capacity in middle layers
mid = num_layers / 2
dist_to_mid = abs(layer_idx - mid) / mid
ratio = 1.0 - dist_to_mid
return int(total_experts * (ratio ** 2))
else: # uniform
return total_experts
# Allocate capacity per layer for 24-layer network with 256 total experts
expert_counts = [
layer_capacity_schedule(i, num_layers=24, total_experts=256, strategy='descending')
for i in range(24)
]
Step 3: Select experts and route tokens within capacity constraints
Route tokens to selected experts, handling overflow when activated experts exceed layer capacity.
def route_with_overflow_handling(logits, active_mask, layer_expert_capacity):
"""Route tokens respecting dynamic activation and layer capacity."""
batch_size, seq_len, num_experts = logits.shape
# Get indices of active experts per token
active_expert_indices = np.where(active_mask)
# Apply load-balancing: if too many experts active, keep top-scoring ones
expert_scores = logits * active_mask - 1e9 * (1 - active_mask)
# For each token, select up to layer_expert_capacity experts
selected_experts = []
for token_idx in range(batch_size * seq_len):
scores = expert_scores.reshape(batch_size * seq_len, -1)[token_idx]
active = np.where(active_mask.reshape(batch_size * seq_len, -1)[token_idx])[0]
# Prioritize by score within active set
active_scores = [(idx, scores[idx]) for idx in active]
active_scores.sort(key=lambda x: x[1], reverse=True)
selected = [idx for idx, _ in active_scores[:layer_expert_capacity]]
selected_experts.append(selected)
return selected_experts
Step 4: Execute sparse forward pass with routing weights
Compute token-to-expert assignments and weighted combination of expert outputs.
# Compute routing weights via softmax over active expert logits
expert_logits_masked = logits * active_mask - 1e9 * (1 - active_mask)
routing_weights = softmax(expert_logits_masked, axis=-1)
# Expert forward pass: dispatch tokens to selected experts
expert_outputs = []
for expert_idx in range(num_experts):
# Collect tokens assigned to this expert
token_mask = (active_mask[:, :, expert_idx] > 0).astype(np.float32)
expert_input = tokens * token_mask.reshape(batch_size, seq_len, 1)
# Expert forward pass (can be parallelized across experts)
expert_output = expert_networks[expert_idx](expert_input)
expert_outputs.append(expert_output)
# Weighted combination of expert outputs
combined_output = np.zeros_like(tokens)
for expert_idx in range(num_experts):
weight = routing_weights[:, :, expert_idx].reshape(batch_size, seq_len, 1)
combined_output += weight * expert_outputs[expert_idx]
Step 5: Backward pass with gradient accumulation across routing decisions
Support gradient flow through both gating network and expert networks to enable end-to-end training.
# Gradient accumulation through routing and expert selection
# (Simplified; real implementation uses automatic differentiation)
# Loss backprop to expert outputs
dL_dexpert_output = loss.backward()
# Gradient through weighted combination
for expert_idx in range(num_experts):
dL_drouting_weight = (dL_dexpert_output * expert_outputs[expert_idx]).sum()
dL_dlogits[:, :, expert_idx] = dL_drouting_weight * (routing_weights - routing_weights**2)
# Load balancing auxiliary loss to prevent expert collapse
auxiliary_loss = load_balance_loss(routing_weights, active_mask)
total_loss = task_loss + 0.01 * auxiliary_loss
Hyperparameter Selection:
When to Use:
When NOT to Use:
Common Pitfalls: