create-model-verification-card

# Create Model Verification Card Create `examples/model_verification_cards/<model-slug>/card.yaml` from public model facts and verified commands. Keep the card small enough for an agent to scan without interpreting logs or reconstructing the execution environment. ## Use the repository resources Treat verification scripts, validators, and launchers as shared infrastructure. Do not modify them merely to make one model card pass. Any such change requires a clear, documented, reusable reason: identify the existing behavior that is insufficient, the affected public workflows or models, why an existing maintained path cannot be used, and add focused backward-compatible tests. Record the justification in the PR description or commit. If the need is model-specific or the reason is not clear, leave the affected verification item unverified instead of adding a card-only workaround. - Validate the result with [scripts/validate_card.py](scripts/validate_card.py). - Verify deterministic HF output with [scripts/verify_hf_inference.py](scripts/verify_hf_inference.py). - Use the inventory and field rules below as the format contract. Do not infer model-specific settings from another family or variant. Do not add a README, evidence blobs, log excerpts, runtime setup, or scheduler metadata to the skill or card. ## Workflow ### 1. Pull facts before drafting Read the model implementation, public HF config, conversion bridge, recipes, tests, examples, and the exact revision being verified. Determine which modes exist; do not infer support from a family name or another model size. Record the public HF model name in commands, its immutable revision in `model.hf_revision`, and the minimum supported Transformers version in `model.min_transformers_version`. Do not introduce an HF snapshot path. Record only two execution-environment facts: the public base container identifier and the exact Bridge commit used for verification. Put them in `verification_environment.base_container` and `verification_environment.bridge_commit`. If the run mounted a checkout over the container, record the mounted checkout commit. Never substitute a private image path for the public base container identifier. The top-level Bridge commit is the default for every item leaf. If one verified item was run from a different clean checkout, put that exact 40-hex commit in the leaf's optional `bridge_commit` field. Omit the field when it would repeat the top-level value, and never use a commit field to disguise uncommitted runtime changes. Items that are not verified must not carry a commit override. ### 2. Create the core inventory and add performance when available Include these twelve required items, even when their status is `unsupported` or `not_applicable`: 1. CPU HF-to-Megatron conversion 2. GPU HF-to-Megatron conversion 3. CPU Megatron-to-HF conversion 4. GPU Megatron-to-HF conversion 5. Manual HF/Megatron forward-pass correlation 6. Deterministic Megatron inference 7. Pretraining 8. SFT 9. SFT checkpoint export and deterministic HF inference 10. Long-context SFT 11. PEFT 12. Checkpoint resume Add `pretrain_performance` as a thirteenth item only when the exact model variant has a canonical public performance recipe. If no such recipe is exported, omit the item instead of adding an unverified placeholder. Once a canonical recipe exists, keep the item in the card even if its run is still unverified. Add `pretrain_fsdp` as an optional hardware-scoped item when a first-class Megatron FSDP performance recipe exists for the exact model variant. Record the FSDP result under `pretrain_fsdp.<hardware>`. Keep it separate from checkpoint-resume and non-FSDP `pretrain_performance` results. Do not embed a baseline, control, computed delta, or relative speedup under another item; agents can derive valid comparisons from the standalone commands, resolved recipes, and raw metrics. The FSDP item does not claim checkpoint save/load unless a separate functional item records that evidence. Add `pretrain_performance_hsdp` as an optional hardware-scoped item when a tuned Hybrid FSDP (HSDP) performance recipe exists for the exact model variant. Use it instead of `pretrain_fsdp` when the recipe shards dense and expert parameters under different strategies, which the single `enabled_features.megatron_fsdp` scalar cannot express. Record the result under `pretrain_performance_hsdp.<hardware>` and mirror it in `verification_index.performance_hsdp`. Keep it separate from `pretrain_performance`: an HSDP recipe usually runs a different convergence contract, so the two are not commensurable. When the same hardware has multiple first-class FSDP runs with different precisions or convergence contracts, use one aggregate hardware container and key its standalone leaves by precision: ```yaml pretrain_fsdp: GB200: status: verified variants: bf16: status: verified precision: bf16 # complete standalone leaf fp8_mx: status: verified precision: fp8_mx # complete standalone leaf ``` The container status is `verified` only when every precision variant is verified; otherwise it is `unverified`. Keep the ordinary direct hardware-leaf shape when there is only one FSDP run. A precision variant repeats its precision as a scalar so agents do not have to infer workload facts from a mapping key. Add `pretrain_weak_scaling` only when at least two completed runs use the same canonical performance recipe, precision, sequence length, maximum step count, and GPUs per node while global batch size grows in direct proportion to total GPU count. Omit it from models without measured scaling data. Put it last in `items`, key it by public hardware, and keep shared provenance on the hardware leaf. Each ordered point records only its GPU count, GBS, public command, and the standard five training metrics: ```yaml pretrain_weak_scaling: GB300: status: verified precision: fp8_mx bridge_commit: 0123456789abcdef0123456789abcdef01234567 last_verified: 2026-08-15 points: - num_gpus: 8 global_batch_size: 512 command: > ./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe example_pretrain_config --mode pretrain --max_steps 50 --seq_length 4096 --global_batch_size 512 metrics: # standard five training metrics expected_result: All points complete with finite metrics and no skipped or NaN iterations. ``` The weak-scaling item is the sole exception that may set global batch size in the public command. It must not override micro batch size. Record raw point metrics only; derive aggregate throughput and scaling efficiency when reading the card rather than storing comparison payloads. A concrete `pretrain_performance.<hardware>` leaf means a canonical performance recipe exists for that hardware, including one explicitly marked **early, unoptimized**. Its item status states whether the card's benchmark run has been verified; an `unverified` leaf still records the existence of the recipe. Existence or functional verification alone does not justify a tuned/optimized claim: require workload-appropriate measured performance and preserve any early, unoptimized label in both the summary and the affected item's `expected_result` so generated pages display it. The library-pretraining readiness targets below are not benchmark-recipe acceptance thresholds. If the card has no concrete performance leaf, start `summary` with this exact disclaimer before the functional-support summary: ```text Performance disclaimer: this card does not record a canonical pretrain performance result; reported timing and throughput metrics are functional verification observations, not standalone optimized performance results. ``` This wording describes the evidence recorded by the card, not whether the model has ever been tuned elsewhere. The validator continues to accept the older untuned-model wording on existing cards for backward compatibility, but new and updated cards must use the wording above. Omit `pretrain_performance` entirely when no canonical recipe exists; do not add an `all` terminal placeholder. Routine timing and throughput metrics in functional pretrain, SFT, PEFT, long-context, and resume items remain sanity observations; they do not become optimized results merely because a separate performance recipe exists. When a tuned recipe exists, scope the summary to the exact `pretrain_performance.<hardware>` leaf. Keep conversion, manual-forward, and base-inference items as direct item records. Key every training item by its public hardware target, and key the dependent SFT export/inference item the same way: ```yaml items: pretrain: H100: status: verified precision: bf16 enabled_features: {} command: ... last_verified: 2026-07-21 metrics: ... expected_result: ... ``` The hardware-scoped names are `pretrain`, `sft`, `sft_export_inference`, `sft_long_context`, `peft`, `checkpoint_resume`, and optional `pretrain_performance`, `pretrain_fsdp`, and `pretrain_weak_scaling`. Use canonical public accelerator identifiers such as `H100`, `B200`, or `GB200`, never a private cluster name. The validator's public-hardware allowlist is authoritative and must be updated when a new accelerator target is introduced. The hardware key replaces the old `gpu_type` field. Each hardware leaf is independent and must carry its own status plus the command or commands, date, metrics, features, and optional commit override that apply to that item. Dependencies resolve within the same hardware key: `checkpoint_resume.H100` consumes `pretrain.H100`, and `sft_export_inference.H100` consumes `sft.H100`. Never fall back across hardware targets. Use the reserved key `all` only as the sole leaf for a model-wide `unsupported` or `not_applicable` limitation. A terminal dependency leaf still names its logical dependency but does not require a matching `all` or concrete-hardware dependency leaf. Use only `unverified`, `verified`, `unsupported`, or `not_applicable`. Do not add `smoke` or an evidence field. #### Add a compact verification index Every card must put a top-level `verification_index` immediately after `summary`. This is a concise directory of the detailed `items`, not a second source of evidence. The validator must reject an index that drifts from the item records. Use `model_level` for the six direct items: the four conversion directions, `manual_forward_pass`, and `inference`. Use `training` for the six functional hardware-scoped items: `pretrain`, `sft`, `sft_export_inference`, `sft_long_context`, `peft`, and `checkpoint_resume`. Keep the optional `pretrain_performance` item separate under `performance`; omit `performance` when the card has no canonical performance recipe. Keep optional `pretrain_fsdp` leaves separate under `fsdp`; omit `fsdp` when the card has no FSDP recipe. Mirror optional `pretrain_weak_scaling` leaves under `weak_scaling`; omit that index when the card has no measured scaling item. Group item names under the same four status names used by the detailed items. For an explicitly indexed hardware target with no corresponding item leaf, summarize the missing verification as `unverified`; do not add empty item leaves or placeholder commands merely to populate the index. Omit empty status buckets and always list every item name explicitly, even when every item in the scope has the same status. Never use the scalar `all` in `verification_index`: an explicit inventory makes it clear which existing items were assessed when new items are added later. For example, a card with partial H100 functional-training coverage and no GB200 functional-training verification uses: ```yaml verification_index: model_level: verified: - hf_to_megatron_cpu - hf_to_megatron_gpu - megatron_to_hf_cpu - megatron_to_hf_gpu - manual_forward_pass - inference training: H100: verified: [sft, sft_export_inference, sft_long_context, peft] unverified: [pretrain, checkpoint_resume] GB200: unverified: [pretrain, sft, sft_export_inference, sft_long_context, peft, checkpoint_resume] ``` When a canonical performance recipe exists, mirror only its concrete leaves: ```yaml performance: H100: verified ``` When an FSDP recipe exists, mirror only its concrete leaves: ```yaml fsdp: GB200: verified ``` When weak-scaling measurements exist, mirror only their concrete leaves: ```yaml weak_scaling: GB300: verified ``` For a multi-variant FSDP hardware container, this scalar mirrors the aggregate container status rather than duplicating the per-precision inventory. Do not add prose comparisons, control payloads, computed deltas, or relative speedups to performance leaves. Agents can compare standalone runs automatically after resolving their recipes. The only exception is a concise warning that names two superficially similar runs that must **not** be compared and the exact convergence-contract differences that make them incompatible. The index may declare an allowlisted public hardware target such as `GB200` before detailed evidence exists. It must also include every concrete hardware target present in the detailed items. Do not use a private cluster name or invent a separate `not_tested` status; `unverified` covers both pending and not-yet-run verification. For `unsupported` and `not_applicable`, leave `command` or `commands`, date, precision, and metrics null, then state the public product limitation in `expected_result`. Put a scalar `precision` on every direct item or hardware leaf. It describes the workload that was actually verified, not every precision the model might support: - for conversion, record the imported or exported weight precision; - for forward pass and inference, record the compute precision; - for training, record the recipe's mixed-precision mode. Use `bf16` for BF16. Training items may instead use `fp8_mx` for MXFP8 or `nvfp4` for NVFP4. Keep MXFP8 and NVFP4 training-only, and do not list either until that exact item has completed in that mode. The outer hardware container of a multi-variant `pretrain_fsdp` item is the only exception: each variant owns the scalar precision and the container owns only aggregate `status` plus `variants`. ### 3. Use the public Slurm launchers Assume the caller supplies the account, partition, concrete runtime image, credentials, and storage mappings outside the card. The top-level `verification_environment.base_container` is provenance only; it is not launcher configuration. Record the portable public launcher and the model-verification workload: - use `scripts/conversion/convert.sh --executor slurm` for CPU and GPU conversion, with portable node and GPU counts; - use `scripts/training/train.sh --nodes ... --gpus-per-node ...` for every training item; - use `scripts/inference/infer.sh --nodes ... --gpus-per-node ... --task ...` for Megatron inference and manual model comparison; select `text-generation`, `legacy-full-prefix-generation`, `vlm-generation`, or `model-comparison` explicitly. The legacy full-prefix task is a slow, non-optimized compatibility path and requires `--legacy-full-prefix`; - invoke public shell launchers directly from the card. Do not create Slurm jobs by calling their Python setup modules, and do not wrap the launchers in `uv run`, `python`, `srun`, or `sbatch`; the shell entry point owns its Python environment and scheduler setup; - use `uv run python ...` only for a local artifact verifier that has no public shell wrapper, such as deterministic Hugging Face inference after export. It must not be used as a substitute for an available shell launcher or to create a cluster job; - use short, ignored repository-relative logical paths under `work/...`; prefer aliases such as `work/data/<dataset>` and `work/cache/<model>` over reproducing a physical storage hierarchy. CPU conversion should omit `--gpus-per-node` when it is genuinely CPU-only. If runtime construction demonstrably requires CUDA even though model weights remain on CPU, request exactly one shared runtime GPU and explain the exception in the item's result. Never request a full GPU node merely to satisfy a launcher or monitor. For example, a portable multi-node VLM verification starts with: ```bash ./scripts/inference/infer.sh --nodes 4 --gpus-per-node 8 \ --task vlm-generation \ --hf_model_path <org>/<model> \ --megatron_model_path work/model-verification/<model>/iter_0000000 \ --image_path docs/images/tp1.png \ --prompt "Describe this image." \ --max_new_tokens 64 ``` The public launchers may read their required generic Slurm configuration from the caller's environment. Do not include `srun`, `sbatch`, concrete account or partition values, container image arguments, `--mount`, `--env`, shell exports, environment-variable references, cluster-specific `--srun-arg` values, or launcher overlays. That wiring is personal to the verifier and does not belong in a model verification card. Never record or reproduce: - execution-environment names, hostnames, IPs, usernames, emails, or accounts; - concrete partitions, reservations, node lists, private image locations, mount sources, or environment forwarding; - host/shared-storage paths, home-directory paths, log locations, or job IDs; - tokens, token-loading commands, private URLs, or private registry references; - environment-specific launcher overlays. Literal chat delimiters (`<|im_start|>`, `<|im_end|>`, `<think>`, and `</think>`) are prompt or output data, not placeholders or storage paths. Loopback IP literals (IPv4 `127.0.0.0/8` and IPv6 `::1`) may identify a single-node inference coordinator; they do not identify private infrastructure. These narrow exceptions do not exempt text enclosed by chat delimiters, other IP addresses, URLs, runtime orchestration flags, or caller-supplied deny terms. Keep private run notes outside the tracked repository and public PR text. If a private codename cannot be recognized generically, pass it to the validator via `--deny-term` or an untracked file through `--denylist "$PRIVATE_DENYLIST"`. The validator reports the match without printing the private term. ### 4. Freeze convergence, execution, and benchmark contracts Before launching any training item, resolve the selected recipe and classify its effective configuration into the three groups below. Do this from the built `ConfigContainer`, not only from command-line overrides. Record the resolved convergence and execution field/value fingerprints in the internal per-model record; keep the public card concise. The **convergence contract** contains settings that change the training objective, examples seen at an optimizer boundary, or numerical update rule: - starting checkpoint and trainable parameter set; - dataset identity/revision, split or bounded selection, sample order, seeds, tokenizer/chat template, truncation, masking, and packing semantics; - sequence length, global batch size, global tokens per optimizer step, total optimizer steps, and total token budget; - objective and loss settings, including label masking, MoE auxiliary/router losses, token dropping/capacity, natural versus forced routing, and loss normalization; - optimizer family, peak/minimum learning rate, schedule shape, warmup and decay horizon, betas, epsilon, weight decay, gradient clipping, and dropout; - model/gradient/optimizer-state precision and loss-scaling behavior; - for PEFT, adapter type, targets, rank, alpha, dropout, and which base weights are frozen. The **execution/performance contract** maps the frozen training semantics to hardware. It may vary across models or be tuned for throughput: - node/GPU count and TP, PP, VP, CP, EP, ETP, DP, and sequence parallelism; - activation recompute, activation/optimizer offload, distributed optimizer or FSDP sharding, checkpoint I/O strategy, and garbage-collection policy; - communication overlap, fused kernels, Transformer Engine implementation, attention backend, CUDA graphs, and compilation; - MoE transport/dispatcher backend such as all-to-all, DeepEP, or HybridEP, provided routing, capacity, token dropping, and auxiliary losses are unchanged. For the final verification-card recipe, no recompute and selective recompute on cheap supported modules are both appropriate starting choices. Choose by measured memory headroom and steady-state performance; selective recompute does not require a no-recompute OOM first. Full recompute is acceptable temporarily for diagnostic bring-up, but can severely reduce throughput. Retain it only when target-scale measurements show lighter choices still fail to fit, or matched runs show better end-to-end throughput with full recompute, for example through larger MBS or lower TP. Keep the workload, GBS, and convergence settings fixed; validate recipe-owned MBS/layout changes in the recipe rather than overriding batch size in the card. Record memory evidence, boundaries/layouts tested, and steady-state performance in the recipe's supporting record; keep public card metrics standalone as required below. Treat micro batch size and gradient-accumulation layout as execution fingerprints. They may be tuned while global batch size, global batch membership/order, loss normalization, optimizer-step boundaries, and total token budget remain intact. Because they can change accumulation order, dropout RNG, MoE token grouping, and auxiliary-loss reduction, require fresh loss sentinels for every layout and do not claim step-by-step numerical parity. Performance settings are **intended** to preserve training semantics, not guaranteed to be bitwise neutral. Parallel reductions, fusions, recompute, and dispatcher implementations can change floating-point order. Every paired run used to claim that a performance-related feature preserves convergence must start from the same weights and use the same clean Bridge commit, dataset and revision, sample order, tokenizer, sequence length, global batch size, token budget, optimizer steps, seeds, objective, routing, precision, optimizer, and learning-rate schedule. Change only the feature under test and unavoidable execution settings. Keep micro batch size and gradient accumulation unchanged when possible; if either changes, require the same loss check but do not claim step-by-step numerical identity. If a verified run deliberately changes any convergence field, make the exact deviation explicit in its command or `expected_result`. Compare LM and auxiliary-loss values at every shared optimizer step. Each candidate value must satisfy `abs(candidate - reference) <= 1e-6 + 0.01 * abs(reference)`. Require finite losses, zero skipped or NaN iterations, and the same qualitative loss trend. If any convergence field differs or any loss falls outside this bound, do not claim that the feature is convergence-neutral. Treat the result as standalone performance evidence, investigate the discrepancy, and explicitly identify the pair as not comparable only when the card would otherwise invite a false comparison. Anything that changes arithmetic precision, forced router balancing, token dropping, packing, or effective batch construction is a convergence change, even when introduced to improve speed. The **benchmark-only configuration** may deliberately change semantics to find an upper throughput bound. It includes mock data, forced MoE load balancing, changed batch/LR or timing-only schedules, and disabled NaN/large-gradient, evaluation, or checkpoint checks. These settings may be valid for a canonical `pretrain_performance` item, but their losses and checkpoints are never convergence evidence. Use `qwen3_30b_a3b_convergence_v2` as the named default cross-model bounded convergence cohort. It retains the optimizer, data, objective, and token-budget contract derived from the resolved Qwen3-30B-A3B H100 recipes while changing offline-packed SFT and PEFT from 2K/GBS32 to 8K/GBS8. The name identifies target settings, not evidence status. Do not call a workload cohort-verified until its recipe owns this contract and a clean-commit run passes the applicable gates below. Its 100 optimizer steps test finite loss, short-horizon loss trend, checkpoint reload, and direct resume; they do not establish full training convergence. Historical `qwen3_30b_a3b_convergence_v1` evidence used 2K/GBS32 for SFT and PEFT. Keep an existing verified card on that exact command, recipe commit, and metrics until the model is rerun. Never relabel 2K evidence as v2 or rewrite its