nvidia-megatron-bridge-perf-moe-long-context

name	nvidia-megatron-bridge-perf-moe-long-context
description	Long-context MoE training guidance for Megatron Bridge. Covers CP sizing, selective recompute, dispatcher choices, and practical patterns from DSV3, Qwen3, and Qwen3-Next long-context experiments.
license	Apache-2.0 AND CC-BY-4.0
metadata	{"when_to_use":"Training MoE at long sequence lengths, or investigating a commit that caused long-context MoE OOM or degraded throughput; 'long context MoE', '128k tokens', 'CP sizing for long sequences', 'selective recompute long context', 'MoE long-context OOM'.","author":"NVIDIA"}

MoE Long-Context Training

Stable docs: @docs/training/moe-optimization.md Card: @skills/nvidia-megatron-bridge-perf-moe-long-context/card.yaml

What Changes At Long Context

Once sequence length moves well past the 4K-class regime, attention memory and activation residency become the dominant constraints. For MoE models, that usually means you need some combination of:

• context parallelism
• selective recompute
• lower precision
• CPU offload for optimizer state
• a dispatcher and PP layout that do not waste the smaller remaining DP budget

Rounded Scaling Patterns

DSV3 on H100

The DSV3 long-context runs show a stable pattern:

• selective recompute works better than full recompute once you move past the shortest contexts
• throughput stays in a fairly narrow band from mid-length through very long contexts if CP is increased appropriately