inference-perf-synthesize

# inference-perf-synthesize ## Purpose The four profiler skills each answer a *different slice* of "where does the time go and why": | Profiler | Skill | Answers | Rigor tier | | --- | --- | --- | --- | | nsys | [`inference-kernel-profile`](../inference-kernel-profile/SKILL.md) | timeline + absolute kernel durations + host-gap structure | L2-ish | | ncu | [`inference-kernel-ncu-profile`](../inference-kernel-ncu-profile/SKILL.md) | per-kernel occupancy / regs / DRAM BW / arithmetic intensity / tensor-core engagement | **L4** | | zymtrace | [`analyze-zymtrace-workload`](../analyze-zymtrace-workload/SKILL.md) | CPU+GPU flamegraph cross-view (sample-share) | L1 | | DCGM | [`inference-dcgm-correlate`](../inference-dcgm-correlate/SKILL.md) | byte-grounded workload-level %SoL over the bench window | L3 | **Do NOT conclude "kernel at-roof / no lever" from SM-busy% (ncu `--set basic` or DCGM SM-active).** An 88-92% SM-busy can hide a kernel at <15% of its FLOP-roofline (persistent/split-K spin). Decide at-roof vs headroom only from ncu `--set full` achieved-TFLOPS vs the FLOP ceiling. Worked example: a DeepSeek-V4-Flash FP8 GEMM read 88-92% SM but was 1-14.5% of FP8 FLOP-SoL -> the synthesized recommendation was "throughput tier", not "kernel rewrite". (Canon: the ncu capture-hygiene section of [`inference-kernel-ncu-profile`](../inference-kernel-ncu-profile/SKILL.md).) Run in isolation they produce four disconnected artifacts. This skill **fuses them into one ranked recommendation ledger**: it reconciles the four views (e.g. zymtrace says category X is hot -> DCGM confirms X is byte-bound -> ncu shows X reaches only Y% of the tensor-core roofline -> nsys shows a host gap before it), turns that into a **ranked list of addressable levers** (config knobs AND concrete vLLM source changes), and **stages each lever as an A/B experiment** so the recommendation is proven on this exact workload before it is believed. It does NOT replace the four skills -- it consumes their outputs. Run the profilers first (or let [`inference-model-optimize`](../inference-model-optimize/SKILL.md) Phase 2 drive them), then run this to synthesize. ## When to use - After profiling a deployment with 2+ of the four profilers, to get a single prioritized "what to change" answer instead of four separate reports. - `inference-model-optimize` Phase 2.5 (diagnose the baseline -> recommend levers that seed the tuning phase) and Phase 8.5 (post-champion: what's left on the table + future-work levers). - When a stakeholder asks "why is it slow and what do we change" and the answer must be data-backed and explainable, not asserted. Do **not** use this skill for: - Capturing a profile -- that is the four profiler skills. This skill reads what they captured. - A which-config sweep -- that is [`inference-tune-sweep`](../inference-tune-sweep/SKILL.md) (this skill *feeds* it the candidate levers). - A single-profiler reading where no cross-view is needed -- just use that profiler's skill. ## Example prompts - "Synthesize the nsys + ncu + zymtrace + DCGM captures into ranked recommendations." - "What should I change in vLLM to make this faster, backed by the profiles?" - "Holistic perf analysis of the gemma4 baseline -- explain the bottleneck with data." - `/inference-perf-synthesize --bundle <evidence-bundle> --focus latency` ## Prerequisites The skill **fails closed** if any of these are not satisfied. 1. An evidence bundle with at least two of the four profiler artifacts present (`*.nsys-rep` / `*.ncu-rep` / zymtrace TSVs / `dcgm_correlation.json`). It will not synthesize from a single source (that is not a cross-view). 2. The capture-validation gates passed for each source (an empty nsys rep is a capture bug, not a finding -- see the nsys capture-validation gate). The skill refuses to draw a conclusion from an unvalidated/empty artifact. 3. For any recommendation it stages as an A/B: a writable experiment-prefixed overlay path + an idle GPU node (plain-K8s GB300 / legacy Slurm-on-K8s) (the A/B itself runs via `inference-tune-sweep` / the controlled-A/B harness). ## Capture orchestration (how to actually get the 4 artifacts) Getting the four artifacts is itself constrained -- these rules are hard-won. Encode them so a capture is right the first time: - **One pod cannot run all four profilers.** `ncu` and the zymtrace CUDA injection (`CUDA_INJECTION64_PATH`) are **mutually exclusive**, `nsys` needs the sidecar image or vLLM `/start_profile`. So a full capture is **2-3 coordinated pods**: (A) the bench pod with the zymtrace injection ON + vLLM torch profiler (`VLLM_TORCH_PROFILER_DIR` + `/start_profile`/`/stop_profile` window), (B) a SEPARATE `nsys` launch-wrap pod (injection cleared, `--cuda-graph-trace=node`), (C) a SEPARATE `ncu` sister pod (`--set basic`, `--kernel-name` scoped). DCGM is correlated POST-HOC over the bench window (no in-pod agent). - **Use a submit-and-queue self-driving Job on a contended cluster** (per `server/AGENTS.md`): the capture pod embeds serve + bench + window-stamp, writes to the RWX PVC, and EXITS to free the GPU, `backoffLimit` re-queues through preemption, guard idempotency on the **result file** (not a marker that could be touched on failure). Make the bench **variance-controlled** (>=3 trials/concurrency, same pod => same node) so the baseline is VERDICT-grade, not a single-shot DRAFT. - **Offline-tokenizer gotcha (L1):** in an `HF_HUB_OFFLINE=1` pod, `vllm bench serve --model <hf-id>` will try to fetch the tokenizer from huggingface.co and fail with `LocalEntryNotFoundError`. ALWAYS pass `--tokenizer <local-weights-dir>` so the bench client uses the on-disk tokenizer. (This silently produced zero results until caught.) ## Workflow ### Phase 1: inventory + validate the captured artifacts Read the bundle. List which of the four profilers produced a *valid* artifact (apply each capture-validation gate). Record the `sol_rigor` available per source. Refuse to proceed if fewer than two valid sources exist. ### Phase 2: reconcile the four views Build a per-kernel-family / per-phase table that joins the sources: - zymtrace sample-share (L1) -> which category/kernel is hot, CPU vs GPU side. - DCGM byte/FLOP (L3) -> is that category byte-bound or compute-bound at the workload level. - ncu (L4) -> for the dominant kernel: achieved roofline %, tensor-core engaged?, occupancy / stall reason / register or smem pressure. - nsys -> absolute duration + whether a host gap precedes it (host-bound vs GPU-bound), cudagraph-captured or not. Flag contradictions (e.g. zymtrace says hot but ncu says near-roofline -> not addressable) -- a contradiction is itself a finding, not something to smooth over. ### Phase 3: emit the ranked recommendation ledger For each addressable lever, write a ledger row: ```text rank | lever | type(config|vllm-src|kernel) | expected_effect | evidence_artifacts[] | source_refs[] | sol_rigor | confidence(DRAFT) ``` `source_refs[]` is MANDATORY for any `vllm-src` / `kernel` lever: the exact vLLM/SGLang branch + commit (resolved to a GitHub URL) and delivery (image|overlay|patchedVllm|infr-patch) from the bundle's `experiment_provenance_v1` block + the campaign's source registry, so a reviewer can open the patch that produced the win. A source-code recommendation with no `source_refs[]` is not actionable. **Code-under-test provenance match:** the `expected_effect` number MUST come from an `evidence_artifacts[]` campaign whose provenance `delivery`+`commit` matches the recommendation's `source_refs[]` -- an `overlay`/offline-prepped campaign cited as the benefit of an `infr-patch` recommendation is a cross-tier DRAFT defect (even when the kernels match). Cite the patch's own run. - **type=config**: a `vllm.extraArgs` / cudagraph / kv-cache-dtype change -> hand to [`inference-tune-sweep`](../inference-tune-sweep/SKILL.md). - **type=vllm-src**: a concrete vLLM code change, staged as a runtime overlay per the delivery ladder (`server/AGENTS.md` "Experiment delivery ladder"): a `subPath` ConfigMap (`overlay_mode: subpath`) for a few files, or the initContainer patch-set (`overlay_mode: patchset-initcontainer`) for a whole patch set on a pullable base -- include the file + the diff sketch + which artifact motivates it. - **type=kernel**: a custom-kernel opportunity -- classify it with the K/R/H/P/A rubric and name the production-representative baseline (a win over a strictly-lower-H/R baseline is a DRAFT, never a VERDICT). Every row MUST list `evidence_artifacts[]` (the exact `.nsys-rep` / `.ncu-rep` / zymtrace TSV / `dcgm_correlation.json` / `kernels.json` paths). A row with no artifact citation is rejected -- this is the explainability contract. ### Phase 4: prove each lever (A/B), do not assert Take the top-ranked levers and stage each as an experiment: - config -> a `campaign_run` cell or a controlled A/B. - vllm-src -> an overlay deploy (experiment-prefixed) + a same-node controlled A/B vs the unpatched arm, **both under matching cudagraph_mode**. Gate each with [`perf-baseline-diff`](../perf-baseline-diff/SKILL.md) (or `perf_baseline_diff`). A lever is promoted from DRAFT (predicted) to VERDICT (proven) only after a same-node + >=3-trial + metric-isolated A/B confirms it. A lever whose A/B comes back within noise is recorded as "tried, null" -- kept in the ledger so the next operator does not re-explore it. ### Phase 5: persist + publish - Write `findings/synthesis-<ts>.md` to the evidence bundle (the ranked ledger + the reconciliation table + the A/B verdicts), via `findings_record`. - Feed the ledger into the perftunereport campaign as a recommendations view so it renders in the PDF and lands in the perf-lake under `campaign=<run-id>` with `focus` + `sol_rigor` recorded. - Record the campaign + `s3://perf-lake/...` paths back into the bundle's `SOURCE.md`/`summary.md`. ## Explainability contract (mandatory) Per `server/AGENTS.md` "All attribution claims must be matched with collected profile data", **every** recommendation and every where/why verdict this skill emits carries: 1. the backing artifact path(s), 2. the `sol_rigor` tier of that evidence (`L1` zymtrace-proxy / `L3` DCGM / `L4` ncu), 3. a DRAFT/VERDICT tier (DRAFT = predicted from profiles. VERDICT = proven by an A/B). A claim missing any of the three is not emitted. An empty/failed capture is a capture bug to fix, never evidence of "unprofilable" -- clear the capture- validation gate first, then conclude. (For zymtrace specifically, an empty-now right after the bench is usually ClickHouse INGEST LAG, not a bug -- wait + requery for the freshest data. See [`server/docs/zymtrace-query-hygiene.md`](../../server/docs/zymtrace-query-hygiene.md).) (One genuine environmental exception seen on GB300: a CUDA 12.x image on a CUDA 13.x driver skews CUPTI -> `CUPTI_ERROR_INVALID_DEVICE` -> 0 kernels for ALL CUPTI clients regardless of hygiene. Grep `CUDA versions. CUPTI/Runtime/Driver` -> use a CUDA-13 image or zymtrace.) ## Verdict rigor (DRAFT vs VERDICT) Synthesis output is **predictive (DRAFT) by construction** -- it ranks levers from profile evidence. A lever becomes a VERDICT only after Phase 4's A/B proves it (same-node, >=3 trials, metric-isolated median TPOT/ITL for latency, against the production-representative baseline). Never report a predicted speedup as achieved. ## Kernel rubric (K/R/H/P/A) Any `type=kernel` recommendation records the candidate AND the named production baseline's `(K,R,H,P,A)` coordinates per `server/AGENTS.md` "Kernel rubric". The H + P proof (tensor-core engagement + roofline %) comes from the ncu artifact, a win over a strictly-lower-H/R baseline is a DRAFT, never a VERDICT. ## Safety - **No silent fallbacks / no fabricated evidence** -- refuses to synthesize from < 2 valid sources or from an unvalidated capture. Rejects any uncited claim. - **A/B before believe** -- a `vllm-src` recommendation is staged as an experiment-prefixed overlay and proven on this workload. It is never applied to a standing deploy or asserted from first principles. - **Experiment isolation + scheduling** -- all A/B arms are experiment-prefixed + `experiment=<id-slug>` labeled and torn down by label. Scheduled per the cluster profile (plain-K8s `default-scheduler` / the Slurm scheduler on legacy Slurm-on-K8s). - **No external posting** of recommended vLLM changes (no upstream PR) without explicit per-turn operator approval. ## Source-of-truth references - The four profiler skills (inputs): [`inference-kernel-profile`](../inference-kernel-profile/SKILL.md), [`inference-kernel-ncu-profile`](../inference-kernel-ncu-profile/SKILL.md), [`analyze-zymtrace-workload`](../analyze-zymtrace-workload/SKILL.md), [`inference-dcgm-correlate`](../inference-dcgm-correlate/SKILL.md). - [`inference-tune-sweep`](../inference-tune-sweep/SKILL.md) + [`inference-perf-tune-report`](../inference-perf-tune-report/SKILL.md) -- where the staged A/Bs run and the recommendations get published. The ranked levers here become the cross-engine variant arms. The proven survivors flow to `perftunereport champion_select` (the baseline-vs-top-X production pick + page 8). - `server/AGENTS.md` -- attribution-must-be-profiled, speed-of-light framing, DRAFT-vs-VERDICT, kernel rubric. ## Full-context reporting (no bare numbers) Per the methodology canon "Every performance number carries its full context (no bare numbers)" (`docs/METHODOLOGY.md`): every number this skill emits MUST carry its full measurement-context descriptor, and every comparison MUST be matched on it. A bare `tok/s` / TPOT / BW / %SoL / speedup is a defect - it cannot set a default, ship a config, or appear in a report. - **Identity:** model (+HF path), hardware (exact ceiling token `GB300`/`B200`), quant, kv-cache dtype. - **Parallelism:** TP, DP (replicas), PP, EP, parallel_strategy. - **Serving cfg:** max-num-seqs, max-num-batched-tokens, gpu-memory-utilization, max-model-len, cudagraph_mode/enforce_eager, async_scheduling, prefix-caching. - **Workload:** dataset, ISL/OSL (or mean in/out tokens), concurrency, num-prompts. - **Regime:** warm vs cold. Latency vs throughput tier. - **Stack:** image/vllm commit, bench backend, serving engine. - **Grounding:** `%SoL` (+ ceiling key from `configs/sol-ceilings.yaml` - never inline a peak), sol_rigor (L1-L4), trials n (mean±std), same-node, baseline named. - **Per-number exact shape (no smoothing):** when reporting more than one number, keep EACH with its own exact shape (ISL/OSL, concurrency, dataset, regime) - never normalize a set to one uniform descriptor that hides per-point variation (e.g. `c=1 @ ISL1024/OSL256` + `c=64 @ ISL4096/OSL512`, NOT one shared "random"). If this skill produces a measurement (tok/s, latency, %SoL, speedup), follow the rigor discipline: capture L1 zymtrace + L3 DCGM (L4 ncu where feasible) Speed-of-Light and publish `--strict`. Canonical map: `server/AGENTS.md` "Rigor discipline index" / `docs/METHODOLOGY.md`. Skills that do not produce measurements are exempt (`server/AGENTS.md` "Speed-of-light framing"). ## Asset validation (review + FAIL LOUD) Every asset this skill emits (findings doc / recommendation ledger / perfreport view) is held to `server/AGENTS.md` "Validate every generated asset"