inference-perf-tune-report

# inference-perf-tune-report ## Purpose Build a multi-page benchmark report PDF -- scatter panels + per-concurrency heatmap tables -- from inference benchmark sweep output. Mirrors the GLM-5.1 reference layout: > **Steady-state window:** the report only renders what the sweep measured. A throughput > sweep cell needs `num_prompts >= 2*c` or its high-c throughput is undercounted > ~1.6-1.8x (ramp/drain-dominated window, `docs/METHODOLOGY.md` trap 4). The > `import_roofline_sweep` importer flags `steady_window_undercount` + WARNs on any cell > `< 2c` - heed it before publishing the report. - **Page 1**: 5 rows (one per `max_num_batched_tokens` in {1024, 2048, 4096, 8192, 16384}) x 2 columns of scatter panes -- left=TTFT vs request throughput, right=tok/s/user vs tok/s/GPU. Inline-labels selected profiling concurrencies (default {1, 8, 32}). Color encodes (hardware, TP). Circle marker = EP, X marker = TP. Hollow markers indicate MTP. - **Page 2**: 3 rows (C in {8, 16, 32}) x 2 columns of heatmap tables showing output tok/s/GPU and TTFT avg (ms) for every (variant, mbt) combo, with gray cells for missing / failed / partial measurements. The renderer is workload-agnostic. It ingests any [`AtlasCell`](../../server/tools/perf_tune_report/schema.py)-shaped JSONL. Backends (vllm-sweep + AIPerf) normalize into the same schema. ## When to use - Comparing inference performance across multiple hardware generations (e.g. H100 FP8 vs B200 NVFP4 vs GB300 NVFP4) for the same model. - Comparing parallelism strategies (TP vs EP) and speculative-decoding variants (MTP on / off) on a single deployment. - Re-rendering a stakeholder-grade report from previously captured benchmark data without re-running the cluster sweep. - Producing a release-day PDF for a new model variant before promoting it to staging / prod. Do **not** use this skill for: - Single-deployment, single-concurrency snapshot benchmarks -- the upstream [`perf-bench`](../inference-perf-bench/SKILL.md) skill is the right tool. Use this skill once you have multiple atlas cells to compare side-by-side. - Training-benchmark reports (step-time / MFU) -- out of scope for this plugin. - Quality / accuracy evaluation -- use [`inference-model-eval`](../inference-model-eval/SKILL.md). ## Example prompts - "Build a perf report comparing Kimi K2.6 on B200 NVFP4 vs GB300 NVFP4." - "Render the atlas PDF for the glm51-multihw campaign." - "Run a glm5p1-style report across H100/B200/GB300 with TP and EP for each." - "Smoke-render the perf-report PDF from the synthetic fixture so I can see the layout before committing cluster time." - "perftunereport campaign_init configs/campaigns/kimi-k26.yaml" - `/inference-perf-tune-report --campaign glm51-multihw` ## Prerequisites 1. **profile-and-optimize bundled MCP server installed**. From a repository checkout, run `bash plugins/profile-and-optimize/server/install.sh`. Client adapter installs can use the server installer in their package cache. The `perftunereport` console script is installed into the same venv as `profile-and-optimize-mcp`. 2. **`matplotlib>=3.8` + `pandas>=2.0`** in the profile-and-optimize venv. Install the optional extra: `.venv/bin/pip install -e '.[perf_tune_report]'`. 3. **For `cell_run` only**: live cluster access (`kubectl` context + namespace + bench pod) AND the same prerequisites as the upstream [`inference-perf-bench`](../inference-perf-bench/SKILL.md) skill (HF_TOKEN, replay-playback dataset access). 4. **No cluster needed** for `campaign_init`, `atlas_aggregate`, `report_render`, or `report_smoke` -- those operate on already-captured data or the bundled synthetic fixture. ## Workflow ### Phase A: smoke-render the layout (no cluster, ~2 seconds) Always do this first when introducing a new operator to the skill. It proves the renderer is healthy and lets the operator see the output shape without committing benchmark time. ```text /inference-perf-tune-report smoke --out /tmp/perftunereport-smoke.pdf ``` or via the MCP tool: ```text perf_tune_report_report_smoke --out /tmp/perftunereport-smoke.pdf ``` The bundled synthetic fixture mirrors the GLM-5.1 PDF coverage exactly: 40 atlas cells, 38 full sweeps, 1 partial, 1 failed, 232 plot-ready points, 20 evicted MTP context cells. ### Phase B: scaffold the campaign Author a campaign YAML under `configs/campaigns/<slug>.yaml` (or any operator-chosen location). Each cell entry must include `cell_id`, `model`, `hardware`, `quant`, `tensor_parallel`, `parallel_strategy`, `mtp`, `max_num_batched_tokens`, `concurrencies`, plus backend-specific extras under `vllm_sweep:` and / or `aiperf:`. ```text perf_tune_report_campaign_init --config configs/campaigns/<slug>.yaml ``` This scaffolds `<server-root>/experiments/artifacts/perf-tune-report/campaigns/<UTC-ts>-<slug>/` with `SOURCE.md`, `summary.md`, frozen `config.yaml`, and empty `cells/` + `commands/` subdirs. Returns the campaign dir in the JSON envelope. ### Phase C: run each cell For each cell in the campaign config, drive `cell_run` with the operator-chosen backend. **This verb is ack-gated (`safety=submits_jobs`)**. The workflow requires the explicit `i_understand_this_submits_jobs=true` parameter (or the CLI flag `--i-understand-this-submits-jobs`) before any actual benchmark runs. `--dry-run` lets you see the generated shell commands without consuming cluster time. ```text # vllm-sweep backend perf_tune_report_cell_run \ --campaign <slug> \ --cell h100-fp8-tp16-ep-mbt4096 \ --backend vllm-sweep \ --i-understand-this-submits-jobs # aiperf backend (same campaign, different cell) perf_tune_report_cell_run \ --campaign <slug> \ --cell b200-nvfp4-tp8-ep-mbt4096 \ --backend aiperf \ --namespace <namespace> \ --bench-pod my-perf-bench \ --kube-context <cluster> \ --endpoint-url http://kimi-k26-svc.<namespace>.svc.cluster.local:8000 \ --served-model moonshotai/Kimi-K2.6 \ --i-understand-this-submits-jobs ``` Each cell writes its raw backend output under `<campaign>/cells/<cell-id>/raw/` and a normalized `<campaign>/cells/<cell-id>/normalized.json` consumed by the next phase. Cell `status.txt` is set to one of `full` / `partial` / `failed` based on whether all requested concurrencies completed. For multi-cell campaigns, loop the operator through each cell -- the sweeps are long-running and benefit from explicit human consent per cell (e.g. checking cluster capacity before each launch). ### Phase D: aggregate into the atlas ```text perf_tune_report_atlas_aggregate --campaign <slug> ``` Unions every cell's `normalized.json` into one `<campaign>/atlas.jsonl` and returns the Coverage summary block that will appear on page 1 of the rendered PDF (e.g. `"Coverage: 40 atlas cells | 38 full sweeps | 1 partial sweeps | 1 failed cells | 232 plot-ready concurrency points"`). ### Phase D1: TPM supported across hardware types (pricing rollup) After aggregation, roll the atlas into a per-hardware **tokens-per-minute (TPM)** capacity table for pricing / capacity discussions. This is a pure post-processing step on already-measured data (no cluster runs). ```text # peak-capacity only: perftunereport tpm_summary --campaign <slug> # add a latency-SLA-bounded operating point (the customer-commitment number): perftunereport tpm_summary --campaign <slug> --ttft-sla-ms 500 --tpot-sla-ms 50 ``` Writes `tpm_summary.{json,csv,md}` to the campaign dir. For each `(model, hardware, quant, TP, strategy, MTP)` group it reports a **peak** point (max output tok/s/GPU -- warm sweep best-case) and, when `--ttft-sla-ms` / `--tpot-sla-ms` are supplied, an **sla** point (highest tok/s/GPU meeting the thresholds), each at **per-GPU / per-replica (=TP GPUs) / per-node** (`--gpus-per-node`, default 8) bases, for both **output-only** and **total** (input+output) TPM. Total-TPM is `n/a` for backends that emit no total-token line. `report_render` also draws a peak-only "TPM supported across hardware types" PDF page, and `publish_to_lake` lands a `tpm_v1` table. Report every TPM number with the warm/cold + ISL/OSL caveat the summary header carries (per `server/AGENTS.md` "Benchmark methodology hygiene"): peak is a warm best-case, the SLA point is the customer-commitment number. **Defaults:** the SLA point + cost columns populate for EVERY campaign without a config block -- `discover_tpm_config` falls back to a default SLA (TTFT<=2000ms / TPOT<=50ms / `gpus_per_node=8`) and a default cost table (representative public on-demand list rates per GPU: H100 $6.16 / H200 $6.31 / B200 $8.60. GB300 unset). Declare a `tpm:`/`cost:` block in the campaign `config.yaml` only to override (per field. Cost is overlaid). `report_render` and `publish_to_lake` read it via `discover_tpm_config`, so all three surfaces emit the same peak + sla points. The SLA point is still left unset (not invented) for a group whose rows lack a decode metric (TPOT/ITL) at/under the SLA -- a throughput-only bench has no decode metric to derive an SLA point from, so re-run the bench with decode capture if SLA-TPM is needed. **Cost + analysis carry-through.** The `cost:` block (`usd_per_gpu_hour: { B200: 4.50, ... }`) overrides/extends the default cost table behind the `$/1M tok` PDF column and the `cost_v1` economics/TCO lake table (`$/1M output|total`, plus `tokens_per_watt` + `power_watts_per_gpu` when DCGM power is captured -- a `cost:` block cannot synthesize tokens-per-watt). Default rates: H100/H200/B200 = representative public list rates, **GB300 = $12.00/GPU-hr ESTIMATE (no public list rate)**. To derive `tokens_per_watt` the cell's DCGM power capture must record BOTH the bench window AND the node (per-node `DCGM_FI_DEV_POWER_USAGE`). A window-only artifact cannot be re-queried later. The atlas also carries per-cell `mean_input_tokens`/`mean_output_tokens` (ISL/OSL, derived from the bench `Total input/generated tokens` lines -> `tpm_v1.mean_isl/mean_osl`), `cache_mode` (declared via `import_perf_bench --cache-mode warm|cold`), and `prefix_cache_hit_rate` (from the bundle metadata). For `tokens_per_watt`, add `power_watts_per_gpu` to the cell's `dcgm_frozen_v1` YAML before `dcgm_correlate`. ### Phase D2: byte-ground every cell to raise sol_rigor (recommended, not a gate) DCGM byte-grounding raises a campaign's `sol_rigor` from `L1` (zymtrace proxy) to `L3` (byte-grounded). The publish CLI is strict by default, so a `dcgm_grounded=false` campaign refuses unless the operator explicitly uses `--no-strict`. That opt-out records the gap on the `campaign_v1` row. For each plot-ready cell, produce a `dcgm_correlation.json` (renderer pages 6 + 6b) via the [`inference-dcgm-correlate`](../inference-dcgm-correlate/SKILL.md) skill (the live Prometheus MCP path) OR the offline CLI verb against a frozen DCGM snapshot: ```text perf_tune_report_dcgm_correlate --campaign <slug> --cell-id <cell> \ --frozen-yaml <dcgm-frozen>.yaml ``` Capture DCGM (SM/DRAM/tensor/GR + NVLINK bytes) concurrently with each cell's bench window so the means are real (see the experiment-isolation rule). A run that skips it can publish at `sol_rigor=L1` only with `--no-strict`. Prefer L3/L4 when DCGM/ncu are available. ### Phase D3: prefill/decode roofline (page 7) - always-on The phase-separated roofline and per-(c,ISL) DCGM utilization answer which concurrency maximizes compute and whether decode approaches the measured HBM ceiling. Supply an operator-captured compatible bundle with a decode concurrency sweep, a prefill ISL sweep, and per-cell DCGM SM, tensor, DRAM, and NVLINK measurements. Then ingest it: ```text perf_tune_report_import_roofline_sweep --campaign <slug> --bundle <out> \ --hardware GB300 --tensor-parallel <tp> --quant NVFP4 --cache-mode cold ``` This writes `cells/<id>-decode` + `<id>-prefill` rows (per-(c,ISL) DCGM + analytical roofline coords in `extra` -> `atlas_v1.extra_json` + the `roofline_v1` lake table) + a `cells/<id>-decode/roofline_sweep.json` (carrying the embedded analytical `ModelShape`) the renderer turns into **page 7**. Page 7 sets `sol_complete` + `dcgm_grounded` + `sol_rigor=L3`. The per-(c,ISL) in-pod `dcgmi` data is what the workload-level `dcgm_correlate` window-mean (Phase D2) cannot produce. Import multiple configs (TP2/TP4/TP8, fp8-KV vs NVFP4-KV, baseline vs optimized, vLLM vs SGLang, ...) into one campaign and page 7 overlays them. **Methodology (`server/tools/perf_tune_report/ROOFLINE-METHODOLOGY.md`):** page-7 panel A plots `x = analytical arithmetic intensity (FLOP/byte)` (from the model's `config.json` via `roofline_math.py`) and `y = flop_per_token x measured tok/s / n_gpus` against PER-GPU ceilings, so a decode point's vertical gap below the memory diagonal IS its HBM utilization. Panel B (Q2) plots BOTH the DCGM `DRAM_ACTIVE` duty-cycle proxy AND the byte-grounded delivered-HBM-BW % (honestly labeled) + the 75% line. For a model not in the built-in registry, pass `--model-config <config.json>` and include the model config in the captured bundle so the analytical axis engages instead of the DCGM-proxy fallback. **Page 7 is now a strict-publish gate:** a throughput/mixed serving campaign with plot-ready points but no page 7 is REFUSED under `publish_to_lake --strict` (the default, `--no-strict` records the gap). The rendered PDF also carries a **"Source under test"** page (vLLM/SGLang commit + delivery + infr patch + GitHub URL) from the bundle's `experiment_provenance_v1` block + `source-registry.yaml`. The campaign's recorded `delivery` is the code-under-test identity: a number from this campaign may be cited as evidence only for THAT delivery, never cross-tier (an `overlay`/offline-prepped run is not evidence for an `infr-patch`, even if the kernels match). ### Phase D4: champion selection (page 8) - the production pick The "what do we ship" synthesis. When the campaign holds a baseline + variant arms (for example, a cross-engine A/B imported via `perf_tune_report_import_variant_ab` from the documented variant bundle layout, with one engine-tagged `vllm-sweep` or `sglang-sweep` cell per arm), `champion_select` ranks the baseline + top-X CROSS-ENGINE under the focus metric + a TPOT SLO, summarizes each across the 4-layer SoL ladder, overlays their rooflines, and emits the production recommendation: ```text