nightly-sync

# Nightly Sync: Main to Dev This skill is read by the automated sync bot during the nightly-sync-main-to-dev workflow. It contains all domain knowledge for merging main into dev, resolving conflicts, iterating on CI, and shipping the PR. --- ## Phase 1: Create the Sync Branch and Merge ### Branch Setup 1. Create branch `$BRANCH` from `origin/dev` 2. Merge: `git merge origin/main --no-edit` 3. Resolve conflicts surgically. Do NOT use global `-X theirs`, and do NOT blanket-checkout main's version of a shared file. Main's version may be taken wholesale only for files in "Files to Override from Main" below, or when you have identified a specific main commit that intentionally removes the dev-only code. For all other conflicts, combine both sides so recent dev-only additions remain present. ### Preserving Dev-Only Additions Do NOT blanket-override all shared files with main's version. Dev has features not yet in main (new classes, new modules, new tests). The merge preserves both sides' non-conflicting additions — only intervene where there is an actual conflict. ### Squash-Merge Chain Detection Dev often develops features as a chain of PRs (PR1 → PR2 → PR3) where each builds on the last. When PR1 is squash-merged to main, git sees main's squashed version and dev's original commits as unrelated changes. A conflict resolution that blindly picks main can silently discard PR2/PR3's improvements on dev. After the merge, check for this pattern: 1. For each conflicted file, run `git log --oneline origin/dev -- <file>` to see if dev has commits that came AFTER the code main is bringing in. 2. If dev has follow-up commits (bug fixes, refactors, extensions), **favor dev's version** for those sections. 3. If the conflict is just main bringing in a clean copy of what dev already has (no follow-ups), main's version is fine. Practical check: run `git diff origin/dev -- <file>` on conflicted files. If dev's code was removed or reverted, investigate whether dev's version is the more evolved one. Real examples from PR #4291: - `emerging_optimizers.py`: Main's version was MORE complete — it squash-merged dev's PRs plus added more. Taking main for that section was correct. - `distrib_optimizer.py`: Main overwrote dev's `GroupedQuantizedTensor` support. Had to restore `_is_distopt_quantized_param` and the expanded `_expand_quantized_param_shard_for_cast` loop while keeping main's NVFP4 additions. This required a surgical merge combining sections from both. Key insight: squash-merge chains can go in EITHER direction. Sometimes main is ahead (it squash-merged dev's work + more), sometimes dev is ahead (it has follow-up PRs). Always diff both ways before deciding which version to favor. Real example from PR #4882 / PR #4318: - `transformer_engine.py`: main had unrelated `TEFusedMLP` refactors, while dev had the new `TEFusedDenseMLP` class. The sync kept the config flag and test but dropped the class and `gpt_layer_specs.py` selection. That is a merge accident: restore the dev class and selection while preserving main's `TEFusedMLP.as_mlp_submodule` refactor. ### Files to Override from Main These files have known semantic conflicts where dev's versions reference args or APIs that main removed or renamed. Take main's version with `git checkout origin/main -- <file>`: - `megatron/training/training.py` — references dev-only args - `megatron/training/initialize.py` — references dev-only args - `megatron/training/utils.py` — references dev-only args - `megatron/training/datasets/data_samplers.py` — references dev-only args - `megatron/core/optimizer/layer_wise_optimizer.py` — constructor signature **Caveat for ALL overrides:** After taking main's version of any file, you MUST run the API Mismatch Detection procedure (see below) on that file. Taking main's caller code while keeping dev's callee implementations is the #1 source of sync bugs. **IMPORTANT: Do NOT take main's `pyproject.toml`, `uv.lock`, or `docker/Dockerfile.ci.dev`.** These three files are a tightly coupled triple — the Dockerfile's `uv sync` command must match the dependency groups in `pyproject.toml`, and `uv.lock` must be consistent with both. Main's versions are missing dev-only dependencies (e.g. `fast-hadamard-transform`, correct TransformerEngine revision) and the `--group no_pypi_wheels` flag needed to install them. Keep dev's versions of all three files. **IMPORTANT: `.github/CODEOWNERS` must NEVER be modified by the sync bot under any circumstances.** Dev's CODEOWNERS is intentionally different from main's — do not take main's version, do not merge them, do not touch the file. If the merge produces a conflict or a non-zero diff against `origin/dev` on this path, restore dev's version verbatim: ``` git checkout origin/dev -- .github/CODEOWNERS ``` Then verify with `git diff origin/dev -- .github/CODEOWNERS` — output must be empty. Modifying CODEOWNERS triggers spurious reviewer requests and conflicts with the dev team's governance; rolling back a CODEOWNERS change after the PR lands is painful. **NEVER manually edit `uv.lock`.** It is a machine-generated lockfile. If it needs to change, it must be regenerated with `uv lock` inside a CUDA container (see `.claude/skills/build-and-test/SKILL.md`). ### Git Source Reconciliation (pyproject.toml) After keeping dev's `pyproject.toml`, check whether main has added NEW git sources to `[tool.uv.sources]` that don't exist in dev's version. Main's merged code may import from packages only available at specific git revisions. 1. Diff the `[tool.uv.sources]` sections: `git show origin/main:pyproject.toml` vs `git show origin/dev:pyproject.toml` 2. For each git source in main but not dev, add it to dev's `pyproject.toml` 3. For sources in both but at different revisions, check whether dev's revision works. If dev's revision is broken (TOML parse errors, missing classes main's code imports), take main's revision instead. Real examples from PR #4291: - `nvidia-resiliency-ext`: Main's `torch.py` imports `get_write_results_queue` which only existed in main's pinned git revision, not on PyPI. Had to add main's git source to dev's pyproject.toml. - `nemo-run`: Dev's pinned revision had a TOML parse error with uv 0.7.2. Had to swap to main's revision. After any changes to `pyproject.toml`, regenerate `uv.lock` inside a CUDA container: ```bash docker run --rm -v $(pwd):/workspace nvcr.io/nvidia/pytorch:26.02-py3 \ bash -c "pip install uv==0.7.2 && cd /workspace && \ uv venv .venv --system-site-packages && uv sync --only-group build && uv lock" # Clean up root-owned .venv: docker run --rm -v $(pwd):/workspace nvcr.io/nvidia/pytorch:26.02-py3 \ bash -c "rm -rf /workspace/.venv" ``` ### API Mismatch Detection (Post-Merge Audit) The merge can create "Frankenstein" code where main's callers use dev's implementations (or vice versa) with different method signatures. This compiles fine but fails at runtime. After the merge, audit cross-boundary call sites: 1. Identify files where main's version was taken (`-X theirs` or explicit `git checkout origin/main`) 2. For each, find all external call sites: classes it instantiates, methods it calls on imported objects, functions from other modules it invokes 3. Verify method names, parameter counts, and signatures match between the caller and the implementation in the merged tree 4. Pay special attention to "interface" modules (files defining base classes) — if main and dev evolved the interface differently, every caller and implementer must agree Real examples from PR #4291: - `multi_latent_attention.py` (main) called `off_interface.group_commit()` but dev's interface only had `group_offload()` — method renamed - `mamba_model.py` (main) called `init_chunk_handler(3 params)` but dev's interface required 6 params — signature expanded on dev - `mamba_model.py` called `mark_not_offloadable()` but dev had `mark_not_offload()` — method renamed - `bulk_offload()` did `.remove()` after `bulk_offload_group()` already `.pop()`d the same item — double-removal from a list Practical detection: ```bash # For each file taken from main, find what it imports and calls grep -rn "from <module> import\|<module>\." megatron/ # Cross-reference with the actual implementations in the merged tree ``` ### File-Specific Merge Lessons These lessons were learned from PR #4291. They may recur if the same files continue to diverge: - `gated_delta_net.py`: If the merge creates code calling non-existent helper methods (e.g. `_resolve_cu_seqlens`), take dev's version wholesale. - `model_chunk_schedule_plan.py`: Watch for missing imports (e.g. `CudaGraphScope`) silently dropped during conflict resolution. - `fine_grained_activation_offload.py`: Critical interface file used by many callers. If main and dev have divergent method names/signatures, prefer dev's implementation and patch main-originated callers to match. - `distrib_optimizer.py`: Dev may have broader type abstractions (e.g. `_is_distopt_quantized_param` covering both FP8 and GroupedQuantizedTensor). Main may simplify to explicit type checks. Restore dev's abstractions. ### Special Handling: data_schedule.py Dynamic CP uses the scheduling implementation from dev: - `BasePackingScheduler`, `DpBalancedScheduler`, `DefaultDynamicCPScheduler`, `wrap_data_iterator`, `get_batch_on_this_rank_for_sequence_packing` (imported by `pretrain_gpt.py` and tests) Do not append the legacy main-only scheduling path. Training and evaluation must use `wrap_data_iterator` so `RerunDataIterator`, PP, and VPP behavior stay on the standard schedules. ### Restore Deleted Files Compare `git ls-tree` between `origin/main` and HEAD to find files in main that are missing from the merged tree. For each: - **Restore** if main's code imports/references it and would break without it (after first checking that the import is still intentional) - **Do NOT restore** if dev intentionally deleted it — check `git log origin/dev -- <file>` for the deletion commit to understand intent - When in doubt, check whether any file in the merged tree imports from the missing file. If nothing imports it, skip it. ### Formatting Run on ALL changed Python files (relative to `origin/dev`), in this order: 1. `black` (version 24, `--config pyproject.toml`) 2. `isort` 3. Order matters: black first, then isort — reverse order can undo isort's work 4. `pylint` on changed `megatron/core/` files — fix missing-docstring and line-too-long violations before pushing ### Pre-push advisory checks Before every `git push` in this workflow (the initial push in Phase 1 AND every fix-push in Phase 3), run these bash checks as guidance. They must never block the push. Review every finding: fix genuine merge accidents and document intentional main removals or formatting/reordering false positives in the PR body. ```bash set +e ( set -euo pipefail MERGE_COMMIT=$(git rev-list --min-parents=2 --max-count=1 HEAD || true) if [ -n "$MERGE_COMMIT" ]; then DEV_REF="${MERGE_COMMIT}^1" MAIN_REF="${MERGE_COMMIT}^2" else DEV_REF="origin/dev" MAIN_REF="origin/main" fi # 1. CODEOWNERS must be identical to dev's. if ! git diff --quiet "$DEV_REF" HEAD -- .github/CODEOWNERS; then echo "WARNING: .github/CODEOWNERS differs from dev. Restore with:" echo " git checkout $DEV_REF -- .github/CODEOWNERS" fi # 2. Dependency-management triple must be identical to dev's. for f in pyproject.toml uv.lock docker/Dockerfile.ci.dev; do if ! git diff --quiet "$DEV_REF" HEAD -- "$f"; then # pyproject.toml is allowed to differ ONLY for git source reconciliation # (new [tool.uv.sources] entries from main). If you intentionally edited # it for that reason, document the reconciliation in the PR body. echo "WARNING: $f differs from dev" fi done # 3. Dev-feature preservation audit. # # The most common sync regression is silently dropping a dev-only feature # that main does not have yet. Pattern: # T0: a feature lands on dev # T1 > T0: the same feature lands on main (possibly reformatted) # The sync runs between T0 and T1. Blindly resolving a conflict in # main's favour drops dev's addition wherever main happened to touch a # nearby line for an unrelated reason. # # For each file the sync touched (modulo skill-sanctioned overrides and # the dependency triple), find every line that satisfies ALL of: # line is on origin/dev (dev had it) # line is NOT on origin/main (main never owned it) # line is NOT in the merged tree (the merge dropped it) # Filter out whitespace-only lines and bracket-only lines (they # frequently differ for cosmetic reasons). # # Files in the "Files to Override from Main" list (training.py, # initialize.py, utils.py, data_samplers.py, layer_wise_optimizer.py) # are exempt by skill convention — main may legitimately win there. # CODEOWNERS and the dep triple are checked above; skip them here. INTENTIONAL_OVERRIDE_REGEX='^(megatron/training/training\.py|megatron/training/initialize\.py|megatron/training/utils\.py|megatron/training/datasets/data_samplers\.py|megatron/core/optimizer/layer_wise_optimizer\.py)$' SKIP_REGEX='^(pyproject\.toml|uv\.lock|docker/Dockerfile\.ci\.dev|\.github/CODEOWNERS)$' FINDINGS=0 for f in $(git diff --name-only "$DEV_REF"..HEAD \ -- '*.py' '*.md' '*.yaml' '*.yml' '*.toml' \ '*.sh' '*.cpp' '*.cu' '*.h' \ | sort -u); do [[ "$f" =~ $SKIP_REGEX ]] && continue [[ "$f" =~ $INTENTIONAL_OVERRIDE_REGEX ]] && continue git cat-file -e "HEAD:$f" 2>/dev/null || continue missing=$(comm -23 \ <(git show "$DEV_REF:$f" 2>/dev/null | sort -u) \ <(git show "$MAIN_REF:$f" 2>/dev/null | sort -u) \ | comm -23 - <(git show "HEAD:$f" 2>/dev/null | sort -u) \ | grep -E '[[:alnum:]_]' \ || true) if [ -n "$missing" ]; then echo "=== $f ===" printf '%s\n' "$missing" FINDINGS=$((FINDINGS + $(printf '%s\n' "$missing" | grep -c .))) fi done if [ "$FINDINGS" -gt 0 ]; then echo "WARNING: $FINDINGS potential dev-only line removal(s) detected. For each:" echo " (a) MAIN INTENTIONALLY REMOVED — find the specific commit in" echo " 'git log origin/main -- <file>' that removed it; document the" echo " SHA in the PR body, then the drop is acceptable." echo " (b) MERGE ACCIDENT — main never explicitly touched that line." echo " RESTORE the dev line (Edit/Write to put it back)." echo "Default to (b); only declare (a) with a specific main commit as evidence." echo "This audit is advisory; continue the push after reviewing the findings." fi echo "nightly-sync pre-push guidance complete" ) GUIDANCE_STATUS=$? if [ "$GUIDANCE_STATUS" -ne 0 ]; then echo "WARNING: nightly-sync pre-push guidance failed with status $GUIDANCE_STATUS; allowing the push to continue." fi exit 0 ``` All pre-push findings are advisory. The hook must return success even when it finds a CODEOWNERS difference, potential dev-feature removal, dependency-triple difference, or an internal audit error. The underlying policies still apply: restore accidental changes, preserve dev-only features, and document exact main commits for intentional removals. A warning by itself is never a reason to stop the workflow or request authorization to continue. Recent regressions the dev-feature audit would have flagged (all "merge accident" type from #4659 and #4716): - `transformer_layer.py` lost `_forward_mlp_router(input_ids=None)` - `token_dispatcher.py` lost the `num_sms_preprocessing_api=...` kwarg on the `_HybridEPManager` call - `moe_layer.py` lost `self._maybe_record_overload_factor(...)` - `gpt_dynamic_inference_with_coordinator.py` lost `from megatron.training.arguments import parse_and_validate_args` - `datasets/readme.md` lost the dev-only "Packing Scheduler" section - PR #4882 / PR #4318 dropped the `TEFusedDenseMLP` implementation and `gpt_layer_specs.py` selection while leaving the config flag and unit test - `data_samplers.py` / `utils.py` / `training.py` kept main's `args.hybrid_context_parallel` instead of dev's `args.dynamic_context_parallel` (counts as a MERGE ACCIDENT — dev's reference is present, main's is the deprecated alias that's False when callers pass `--dynamic-context-parallel`). These files are on the override list so the audit treats them as "advisory", but you should still rename `args.hybrid_context_parallel` →