메뉴
MoE expert-parallel communication overlap in Megatron Bridge. Use when the user asks about overlap_moe_expert_parallel_comm, MoE dispatch overlap, flex dispatcher, DeepEP overlap, or expert wgrad scheduling.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | moe-comm-overlap |
| description | MoE expert-parallel communication overlap in Megatron Bridge. Use when the user asks about overlap_moe_expert_parallel_comm, MoE dispatch overlap, flex dispatcher, DeepEP overlap, or expert wgrad scheduling. |
For what MoE communication overlap is and when to use it, see:
docs/training/communication-overlap.mdcard.yaml (co-located)cfg.comm_overlap.overlap_moe_expert_parallel_comm = True
# Optional: delayed wgrad for additional overlap
cfg.model.moe_delay_wgrad_compute = True
# IMPORTANT: disable shared expert overlap when using dispatch overlap
cfg.model.moe_shared_expert_overlap = False
expert_model_parallel_size > 1num_moe_experts > 1moe_token_dispatcher_type must be "alltoall" or "flex"virtual_pipeline_model_parallel_size) must be set (non-None)Setting moe_flex_dispatcher_backend alone does not activate flex dispatch.
You must also set moe_token_dispatcher_type = "flex".
src/megatron/bridge/training/comm_overlap.pysrc/megatron/bridge/training/flex_dispatcher_backend.pysrc/megatron/bridge/training/config.pytests/unit_tests/training/test_comm_overlap.pytests/unit_tests/training/test_deepep.pyShared expert overlap conflict: moe_shared_expert_overlap and
overlap_moe_expert_parallel_comm can conflict. Disable shared expert
overlap when using the dispatch overlap path.
PP without VPP: MoE overlap requires VPP when pipeline parallelism is active. Without it, the overlap scheduling cannot interleave correctly.
Flex != backend flag: moe_flex_dispatcher_backend="deepep" alone
does nothing if moe_token_dispatcher_type is still "alltoall".
Conservative recipe defaults: Most public recipes leave MoE overlap disabled. You need to explicitly enable it via overrides.
Look for overlap-related log messages during initialization. The comm overlap
validation in comm_overlap.py will raise if prerequisites are not met, so a
clean startup confirms the feature is active.
Guide for adding support for new LLM or VLM models in Megatron-Bridge. Covers bridge, provider, recipe, tests, docs, and examples. Use when the user asks to add, support, onboard, or integrate a new model, or when creating bridges, providers, or recipes for a new model family.
Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation. Use when creating Slurm scripts, scaling to multi-node, or debugging multi-node job failures.
Developer environment setup, CI/CD workflows, and CI failure debugging for Megatron Bridge. Covers container-based development, uv package management, pre-commit hooks, running tests, CI failure investigation, and common pitfalls. Use when onboarding, setting up a dev environment, troubleshooting build issues, or investigating CI failures.
Code style and quality guidelines for Megatron Bridge. Covers naming, type hints, ruff enforcement, keyword-arg safety, copyright headers, logging, and common anti-patterns. Auto-invoked during code review and when writing new code.
Structured framework for verifying numerical parity of HF<->MCore weight conversions. References existing tools and the add-model-support skill. Use when debugging weight mismatches, verifying checkpoint round-trips, or choosing which verification tool to run.
Validate and use MoE expert-parallel communication overlap in Megatron-Bridge, including overlap_moe_expert_parallel_comm, delay_wgrad_compute, and flex dispatcher backends such as DeepEP and HybridEP.