inference-model-optimize

# inference-model-optimize > **Fast model loading (always-applied):** when standing up a vLLM deploy, never load 100s-of-GB single-stream via s3fs FUSE (~50 min for a large model). Prefer, in order: a fast-model-loading endpoint when reachable -> parallel multipart to local NVMe (the template's `stage-model-parallel.py`) -> `runai_model_streamer` (`--load-format runai_streamer`) -> tensorizer. Flag a slow load loudly: if effective rate < ~500 MB/s on a large model, STOP and switch. Details: [`server/docs/inference-fast-model-loading.md`](/plugins/profile-and-optimize/server/docs/inference-fast-model-loading.md). ## Purpose One orchestrator for the question "given a new model, how do we find its best inference performance on a B200 / GB300 fleet?" It does **not** re-implement any phase -- it sequences the mature per-phase skills behind a single, gate-driven workflow and one evidence bundle, so an operator drives the whole profile -> tune -> calibrate -> quantize -> validate -> spec-decode -> bench -> publish loop without hand-stitching seven skills together. The orchestrator owns three things the individual skills do not: 1. **Phase sequencing + gates** -- each phase has an explicit go/no-go. A red pauses the pipeline and reports, never auto-advances. 2. **One experiment-id end to end** -- the run-id is the join key across the harness bundle, every cluster object's `experiment=<id-slug>` label, and the perf-lake `campaign=<id>` rows. 3. **Discipline enforcement** -- every number defaults to **DRAFT**. The run always publishes to the perf-lake with a recorded `focus` + `sol_rigor`. ## When to use - A model the team has never deployed lands (new HF id) and you want its best serving config + a published roofline, not just "does it boot". - You want to re-baseline an existing model through the full pipeline after a vLLM bump or a new GPU generation. - You want a single reproducible record (one evidence bundle) that ties the deploy config, profiles, quant recipe, draft head, and final numbers together. Do **not** use this skill for: - Promoting a model to **staging / prod** -- that is a separate onboarding track (deploy CLI + QA suite). This harness stops at a lab champion. - A single one-off benchmark of an already-deployed model -- use [`inference-perf-bench`](/plugins/profile-and-optimize/skills/inference-perf-bench/SKILL.md) directly. - Any individual phase in isolation -- call that phase's skill directly (they are all listed under "Phases" below). ## Example prompts - "Optimize MiniMax-M2.7 for inference end-to-end on B200." - "Bring up `zai-org/GLM-5.2` through the full pipeline and publish the roofline." - "Find the best perf for this new model: profile, tune, quantize, train a draft." - `/inference-model-optimize --model minimax-m27 --hf-id MiniMaxAI/MiniMax-M2.7 --tp 8 --gpu b200` ## Prerequisites The skill **fails closed** if any of these are not satisfied. 1. The profile-and-optimize bundled MCP server is installed (the `perftunereport` console script + `perf_baseline_*` / `evidence_*` verbs are reachable). 2. The model-bringup template workspace is checked out alongside this repo (the `scaffold-model-bringup.sh` scaffolder and the per-model bring-up template live there). 3. Live cluster access + the right **cluster profile** (`scaffold-model-bringup.sh --cluster-profile`): **`k8s`** = plain Kubernetes (GB300: `default-scheduler`, k8s `nvidia.com/gpu` authoritative, NO storageclass -> emptyDir + hf-pull, artifacts -> S3), **`slurm`** = Slurm-on-K8s (legacy B200: the Slurm scheduler pod, VAST PVC). Detect: a `slurm-controller` + `slurm.example.com/lock` taint => `slurm`, else `k8s`. 4. Free GPUs confirmed BEFORE any GPU phase: on **k8s** via `kubectl get nodes -l node.kubernetes.io/instance-type=<gpu>,cluster.example.com/state=dev` + k8s GPU requests. On **slurm** via `sinfo`/`squeue`. GB300 node = 4 GPU (TP8 spans 2 nodes via the NVLink domain). B200 = 8 GPU. 5. For the quant + spec-decode phases: a non-NVFP4 base checkpoint (BF16 or FP8) is reachable on HF/S3 (those phases consume the base, not the serving NVFP4). ## Interaction style Iterative and gate-driven: run one phase, report the gate verdict + the headline number (DRAFT unless promoted), then ask before advancing. Never auto-advance past a red gate. The operator may start at any phase (`--start-phase`) or skip phases that do not apply (e.g. skip Phase 6/7 for a model with no draft-head plan). ## Workflow ### Phase 0: resolve intent + scaffold Resolve the operator's request to concrete parameters and state them back: - `model-slug` (short, no quant suffix, e.g. `minimax-m27`), `hf-id` (base, e.g. `MiniMaxAI/MiniMax-M2.7`), optional `hf-nvfp4-id` (a pre-quantized variant if one already exists), `tp`, `gpu-type` (`b200` | `gb300`), `quant-target` (`nvfp4` | `fp8` | `none`), `spec-method` (`eagle3` | `deepseek_mtp` | `dflash` | `ngram` | `none`). - Read the model card / `config.json` for arch family, attention type (MLA needs `block-size: 64`), `num_hidden_layers`, `max_position_embeddings`, parsers -- the same step-1 discovery a production onboarding does. Then scaffold the per-model harness bundle + evidence bundle (one shell call, no GPU spend): ```text scaffold-model-bringup.sh \ --model-slug minimax-m27 \ --hf-id MiniMaxAI/MiniMax-M2.7 --hf-serve-id nvidia/MiniMax-M2.7-NVFP4 \ --tp 4 --gpu-type gb300 \ --cluster-profile k8s --ns <namespace> \ --quant-target none --spec-method deepseek_mtp ``` (Legacy Slurm-on-K8s/B200 form: `--tp 8 --gpu-type b200 --cluster-profile slurm`.) This renders the per-model harness bundle (03a PVC, 03b puller, 03c deploy, `my-values-minimax-m27.yaml`, `profiling/`, `quantize/`, `spec-decode/`) and an evidence bundle at `experiments/artifacts/model-optimize/<run-id>/` with `SOURCE.md` + `summary.md`. **The run-id printed here is the experiment-id** -- record it. Every later phase uses it as the `experiment=<id-slug>` label and the `campaign=<id>` key. Report the scaffolded paths. Ask before deploying. ### Phase 1: deploy baseline (cluster-profile aware) **First, resolve the loader (the single source of truth, not a guess).** Run the loader advisor to pick the weight loader for this tier (decision tree + measured basis in [`server/docs/inference-fast-model-loading.md`](/plugins/profile-and-optimize/server/docs/inference-fast-model-loading.md) "Loader selection for a serving tier"): `python3 tools/loader_advisor.py --serve-args "<the tier's vllm serve args>" --hf-egress yes|no --image-has-runai yes|no --object-store yes|no --emit experiments/artifacts/model-optimize/<run-id>/`. It writes `loader_advisor.json` + `LOADER.md` (recommended loader + tier + per-gate pass/fail + rationale). Record the recommended loader + rationale in the bundle `SOURCE.md`, then drive the scaffolder with it (`scaffold-model-bringup.sh --loader auto|hf-pull|runai`) so the rendered baseline matches. Rule of thumb the advisor encodes: **non-MTP tier -> RunAI streamer** (streams weights from object storage straight to GPU, fastest, no HF dependency / emptyDir / FUSE), **MTP tier -> hf-pull** (MTP-native. RunAI needs the drafter patch + double-streams). **Then resolve the TP (right-size to active params, do not over-shard).** Run the TP right-size advisor to size tensor-parallel to the model's ACTIVE params, not its total -- a small-active MoE served at a TP higher than memory requires spreads ~3B active over N near-idle GPUs (the over-provisioning trap: measured **~2.7x worse tok/s/GPU** plus flat-power OPEX waste on the 30B-A3B class): `python3 tools/tp_rightsize_advisor.py --total-params-b <T> --active-params-b <A> --dtype nvfp4|fp8|bf16 --current-tp <TP> --emit experiments/artifacts/model-optimize/<run-id>/`. It writes `tp_rightsize.json` + `TP-RIGHTSIZE.md` (recommended TP + over-provisioned flag + ~$/replica/month at matched throughput + `measured|extrapolated` confidence). Rule it encodes: **recommend the lowest TP that fits weights in memory while keeping active >= ~3B/GPU**. It flags OVER-PROVISIONED only when active/current_TP < 3B (so large-active tiers -- e.g. a 13B-active model at TP4 -- are NOT touched). `extrapolated` (total > ~40B) = verify with a same-node TP A/B before shipping. Record the recommended TP + rationale in `SOURCE.md` and deploy at that TP. Preflight a free GPU node, then deploy the baseline (BF16 or vendor FP8/NVFP4 if one exists) via the scaffolded bundle's `03a -> 03b -> 03c`. - **plain-K8s GB300 (the DEFAULT):** `schedulerName: default-scheduler` + `resources.requests.nvidia.com/gpu` (k8s device-plugin is authoritative). Preflight `kubectl get nodes -l node.kubernetes.io/instance-type=<gpu>,cluster.example.com/state=dev` + k8s GPU requests==0. NO storageclass -> emptyDir + hf-pull. GB300 node = 4 GPU (TP8 spans 2 nodes via the NVLink domain). - **legacy Slurm-on-K8s B200 (only if you detect a `slurm-controller` + `slurm.example.com/lock` taint):** `schedulerName: <slurm-scheduler-pod>` + the lock toleration. Preflight `sinfo -N -o '%N %t %G'` / `squeue`. NEVER `default-scheduler` + a hard `nodeSelector` GPU grab on Slurm-on-K8s (it double-books GPUs Slurm has already allocated). The deploy carries the zymtrace CUDA injection by default (copied from the bring-up template's `profiling/experiment-deploy-template.yaml`). Delivery: for a modified-vLLM arm, default to the runtime overlay ladder -- a subPath ConfigMap, or the initContainer patch-set (the template's `overlay-patchset.sh`) for a whole engine patch set on a pullable base image. Bake an image only for prod / compiled changes. Gate: pod Ready + a curl smoke + `verify-experiment-labels.sh <id-slug> --post` returns PASS. Record the baseline as a DRAFT perf-baseline. ### Phase 2: profile the baseline Drive the profiling skills against the live pod (all on by default in the template). Sequence: - [`inference-decode-step-budget`](/plugins/profile-and-optimize/skills/inference-decode-step-budget/SKILL.md) at c=1..8 -- is the hot path kernel-bound / host-bound / comm-bound? - [`analyze-zymtrace-workload`](/plugins/profile-and-optimize/skills/analyze-zymtrace-workload/SKILL.md) -- GPU+CPU cross-view. Which kernel family dominates. - [`inference-kernel-profile`](/plugins/profile-and-optimize/skills/inference-kernel-profile/SKILL.md) (nsys) and, when a per-kernel which-kernel verdict is needed, [`inference-kernel-ncu-profile`](/plugins/profile-and-optimize/skills/inference-kernel-ncu-profile/SKILL.md) (ncu). - [`inference-dcgm-correlate`](/plugins/profile-and-optimize/skills/inference-dcgm-correlate/SKILL.md) for byte-grounded workload %SoL (raises sol_rigor to L3). - [`inference-graph-diff`](/plugins/profile-and-optimize/skills/inference-graph-diff/SKILL.md) only when comparing two compile configs. Gate: a profile actually captured (the nsys capture-validation gate must be clear -- an empty rep is a capture bug, not a verdict). Capture all four where a where/why verdict matters. The next phase fuses them. **Capture orchestration** (one pod can't run all four. Use a submit-and-queue self-driving Job with a variance-controlled bench >=3 trials/concurrency. Pass `--tokenizer <local>` in an offline pod) is documented in [`inference-perf-synthesize`](/plugins/profile-and-optimize/skills/inference-perf-synthesize/SKILL.md) "Capture orchestration" -- reuse the `experiments/` Job templates the scaffolder renders. ### Phase 2.5: synthesize the profiles into ranked recommendations Hand off to [`inference-perf-synthesize`](/plugins/profile-and-optimize/skills/inference-perf-synthesize/SKILL.md): fuse the four profiler artifacts (nsys + ncu + zymtrace + DCGM) into ONE ranked, artifact-cited recommendation ledger (config knobs AND concrete vLLM-source overlays). Every row cites its backing artifact + `sol_rigor` (the explainability contract). Every row is DRAFT (predicted) until an A/B proves it. The top-ranked levers seed Phase 3, and any `vllm-src` lever becomes an overlay A/B there. ### Phase 3: tune CROSS-ENGINE (vLLM + SGLang) via the variant A/B The tuning spine is the **engine-agnostic variant A/B** -- we no longer pick an engine by reputation. Every candidate config (vLLM AND SGLang) is an arm proven on ONE shared bench client so the numbers are comparable by construction. Hand off to [`inference-tune-sweep`](/plugins/profile-and-optimize/skills/inference-tune-sweep/SKILL.md), which fills the rendered bundle's `variants/arms.tsv` with both engines' levers (one arm per lever. First row per engine = that engine's baseline) and runs the same-node, >=3-trial sweep: ```text run-variant-ab.sh readback arms-crossengine.tsv <out> # vLLM + SGLang arms, c-sweep, SoL/arm ``` - **vLLM levers:** `max_num_batched_tokens`, `kv-cache-dtype`, `cudagraph_mode`, `--enable-expert-parallel`, MoE backend. Plus any Phase-2.5 `vllm-src` overlay. - **SGLang levers:** `--moe-runner-backend`, `--attention-backend`, cuda-graph, `--mem-fraction-static`, `--disable-radix-cache` (match caching across engines BEFORE crowning a throughput champion -- a radix-cache replay can manufacture a phantom win). Then `perftunereport import_variant_ab --bundle <out> --model <id>` (engine-tagged `vllm-sweep` / `sglang-sweep` cells) -> `atlas_aggregate`. For any decode-latency ship/no-ship claim, confirm with the same-node controlled A/B (`run-controlled-ab.sh`. SGLang via `BENCH_TARGET=deploy/<client> BACKEND=openai`). Gate: a champion that beats the Phase-1 baseline on the run's `focus` metric (throughput tok/s/GPU OR median TPOT/ITL -- never output tok/s at small `num_prompts` for a latency claim). Promote DRAFT -> VERDICT only same-node + >=3 trials. The cross-engine ranking + the production pick happen in Phase 8.4 (`champion_select`). Record the per-engine top-3 here. ### Phase 4: calibrate + quantize Only if `--quant-target` != `none`. Hand off to [`inference-quantize-calibrate`](/plugins/profile-and-optimize/skills/inference-quantize-calibrate/SKILL.md): a calibration-dataset Job + a quantize Job (NVIDIA ModelOpt -> NVFP4 by default, llm-compressor -> FP8 alternate) that writes weights + per-layer KV/activation scales to an experiment-prefixed PVC. Gate: the quantize Job completes and the output `config.json` declares the expected quant method. This phase is