inference-perf-baseline-bridge

# inference-perf-baseline-bridge ## Purpose Translate the output of an [`inference-perf-bench`](/plugins/profile-and-optimize/skills/inference-perf-bench/SKILL.md) (or its colloquial alias `ai-bench`) run into the shape the workload-agnostic [`perf-baseline-record`](/plugins/profile-and-optimize/skills/perf-baseline-record/SKILL.md) / [`perf-baseline-diff`](/plugins/profile-and-optimize/skills/perf-baseline-diff/SKILL.md) registry expects. The bridge fixes three pain points: 1. The upstream `perf-bench` skill's report is a markdown table per model - not a JSON payload the registry can hash and diff. 2. The metric set (TTFT p50/p95/p99, ITL p50, tok/s, tok/s/user, request latency p50, prefix cache hit rate delta, GPU cache utilization peak) needs a stable schema name so future diffs find apples-to-apples baselines. 3. Per-metric regression tolerances differ - TTFT regressions of 5% are usually real, throughput regressions of 3% may be capacity contention, prefix cache hit rate movements should be expressed as absolute percentage points (not percent-of-percent). This bridge does NOT add a new MCP verb. It wraps the existing `mcp__profile_and_optimize__perf_baseline_record` and `mcp__profile_and_optimize__perf_baseline_diff` verbs with a fixed schema kind (`inference_perfbench_v1`) and operator-tunable per-metric tolerances. ## Schema: `inference_perfbench_v1` ```json { "schema": "inference_perfbench_v1", "model": "<served-model-name>", "model_path": "<HF-org/repo>", "deployment": { "namespace": "<k8s-namespace>", "cluster": "<kube-context>", "service": "<svc-name>", "vllm_version": "<v0.NN.M>", "tensor_parallel_size": <int>, "quantization": "<NVFP4 | FP8 | BF16 | FP4 | INT4>", "kv_cache_dtype": "<fp8_e4m3 | bf16 | ...>", "speculative_decoding": "<eagle3 | mtp | none>" }, "workload": { "dataset": "<hf-org>/replay-playback", "split": "<2025_07 | 2025_04 | 2025_01 | all>", "conversation_num": <int>, "concurrency_sweep": [<int>, ...], "input_token_band": "<1k | 10k | 100k | mixed>", "output_token_band": "<1k | 1.5k | 2k | mixed>" }, "metrics": { "<concurrency>": { "throughput_toks_per_s": <float>, "ttft_p50_ms": <float>, "ttft_p95_ms": <float>, "ttft_p99_ms": <float>, "itl_p50_ms": <float>, "tps_per_user": <float>, "request_latency_p50_ms": <float>, "errors": <int> } }, "server_metrics_delta": { "prefix_cache_hit_rate_pre": <float>, "prefix_cache_hit_rate_post": <float>, "gpu_cache_usage_perc_peak": <float>, "num_requests_running_peak": <int>, "avg_generation_throughput_toks_per_s": <float> }, "kernel_class_gpu_pct": { "fp8_bmm": <float|null>, "tp_all_reduce": <float|null>, "cuda_event_sync": <float|null>, "mla_attention": <float|null>, "moe_routing_finalize": <float|null>, "per_token_group_quant": <float|null>, "<other-class>": <float|null> }, "cpu_spinpoll_pct": <float|null> } ``` ## Optional fields The two top-level fields `kernel_class_gpu_pct` and `cpu_spinpoll_pct` are **optional**. They surface profile-grade signals that the upstream `perf-bench` runbook does not produce on its own - they are populated when the operator also captures a [zymtrace](/plugins/profile-and-optimize/skills/zymtrace-anchored-query/SKILL.md) or Nsight-Systems profile against the same time window. Field shapes and origins: - **`kernel_class_gpu_pct`** is an **open map** of kernel-class name to percent of GPU time. The canonical class set is the one typically observed in a plain-FP8 MoE serving profile: `fp8_bmm`, `tp_all_reduce`, `cuda_event_sync`, `mla_attention`, `moe_routing_finalize`, `per_token_group_quant`. Classes outside this set go under arbitrary `<other-class>` keys (free-form). The map is derived from a GPU-side `event_kind = 'cuda'` zymtrace query grouped by kernel-name regex. The source bucket-rules live in the [zymtrace-anchored-query](/plugins/profile-and-optimize/skills/zymtrace-anchored-query/SKILL.md) skill's "Kernel-class bucketing" appendix. - **`cpu_spinpoll_pct`** is a single scalar: the percent of CPU on-cpu samples that fall inside vLLM's spin-poll IPC path (`shm_broadcast.py::{acquire_read,wait,memory_fence,timeout_ms,should_warn,dequeue}` + `utils.py::sched_yield`). Populated from the same zymtrace bundle's CPU-side `event_kind = 'on_cpu'` query. **Diff semantics on null**: when either side of a per-metric comparison is `null`, the bridge skips that metric in `Phase 2: per-metric diff with tolerances` and emits a `NULL_LEGACY_BASELINE` row in the diff output so the operator can see which fields were unmeasured. This keeps existing `inference_perfbench_v1` baselines registered before these fields existed diff-compatible with new bundles that include them - the registry never breaks. When a true breaking change is needed (e.g. metric-shape restructure of `metrics.<concurrency>`), the schema name will bump to `inference_perfbench_v2`. Additive-only changes stay on `_v1`. ## Default per-metric tolerances | Metric | Default tolerance | Direction | Rationale | | --- | --- | --- | --- | | `throughput_toks_per_s` | 3% | regression = current < baseline | Throughput is the headline metric. Small regressions are real | | `ttft_p50_ms` | 5% | regression = current > baseline | TTFT is sensitive to scheduler / cache cold-start, 5% is the usual signal floor | | `ttft_p95_ms` | 10% | regression = current > baseline | Tail latency wider band - outlier-prone | | `ttft_p99_ms` | 15% | regression = current > baseline | Same | | `itl_p50_ms` | 3% | regression = current > baseline | ITL is decode-bound. Small regressions are real | | `tps_per_user` | 3% | regression = current < baseline | Per-user throughput. Same band as overall throughput | | `request_latency_p50_ms` | 5% | regression = current > baseline | Same as TTFT p50 | | `prefix_cache_hit_rate_post` | 2 absolute pts | regression = current < baseline | Cache hit rate moves in absolute pts, not percent-of-percent | | `gpu_cache_usage_perc_peak` | 5 absolute pts | regression = current > baseline | Capacity ceiling proxy | | `kernel_class_gpu_pct.<any>` | 3 absolute pts | regression = current > baseline | Kernel-class %-of-GPU drift, 3pp is the usual signal floor for hot kernels (e.g. TP all-reduce drifting from an expected ~10% to an observed ~15% is an actionable 5pp delta) | | `cpu_spinpoll_pct` | 5 absolute pts | regression = current > baseline | CPU spin-poll waste, >5pp jump indicates `VLLM_USE_SHM_BROADCAST_BLOCKING=1` regressed off, or a new vLLM release re-introduced busy-wait IPC | Operator can override any of these via `--tolerance.<metric> <value>` on the command line. The two optional fields' tolerances are only evaluated when both sides of the diff have non-null values. Otherwise the metric is reported as `NULL_LEGACY_BASELINE` and skipped (see "Optional fields" above). ## When to use - After an [`inference-perf-bench`](/plugins/profile-and-optimize/skills/inference-perf-bench/SKILL.md) run completes (Phase 9 of the upstream runbook). - Periodic regression sweep - weekly cron diffs each registered baseline against the most recent run. - Pre-promotion gate - block staging or prod promotion if the diff verdict is RED. - Pairing perf vs quality - register the baseline with `--notes "GPQA=<score>. MMLU-Pro=<score>"` from the [`inference-model-eval`](/plugins/profile-and-optimize/skills/inference-model-eval/SKILL.md) run so future diffs can confirm the regression is not masked by a quality gain (or vice versa). Do **not** use this skill for: - MLPerf training step-time / MFU baselines - use [`perf-baseline-record`](/plugins/profile-and-optimize/skills/perf-baseline-record/SKILL.md) directly with `--family <bench>` and a custom schema. - One-off diagnostic measurements - register a baseline only when the measurement is worth remembering. ## Example prompts - "Register the kimi-k25 perf-bench result as the new baseline." - "Diff this morning's deepseek-v4 ai-bench against last week's baseline." - "Inference perf regression check on glm-5-fp8 with tolerance.tps=5%." - `/inference-perf-baseline-bridge record --model kimi-k25 --source experiments/artifacts/inference-perf-bench/<run-id>/` - `/inference-perf-baseline-bridge diff --baseline experiments/artifacts/perf-baselines/inference/kimi-k25/2026-05-19T20:00:00Z/ --current experiments/artifacts/inference-perf-bench/<run-id>/` ## Prerequisites 1. **Source bundle path** - `--source <path>` pointing at an `experiments/artifacts/inference-perf-bench/<run-id>/` directory. 2. **Model name** - `--model <served-model-name>` (matches the value returned by the model's `/v1/models` endpoint). 3. **`PROFILE_AND_OPTIMIZE_REPO_ROOT`** for the registry path. ## Interaction style Mostly autonomous. One pause: confirm the per-metric tolerances and the registry path before write (record) or before computing the diff verdict (diff). ## Workflow ### Mode 1: record #### Phase 1: parse the perf-bench bundle into the schema shape Read the upstream perf-bench output: - `experiments/artifacts/inference-perf-bench/<run-id>/perf-bench-report-*.md` - parse the per-concurrency rows. - `experiments/artifacts/inference-perf-bench/<run-id>/aiperf-c<N>.log` - extract the headline metrics per concurrency. - `experiments/artifacts/inference-perf-bench/<run-id>/<model>-metrics-pre.prom` and `<model>-metrics-post.prom` - extract `vllm:prefix_cache_hit_rate`, `vllm:gpu_cache_usage_perc`, `vllm:num_requests_running`, `vllm:avg_generation_throughput_toks_per_s`. If the operator also captured a paired profile bundle (zymtrace or Nsight) against the same window, source the two optional fields: - `experiments/artifacts/zymtrace-bundles/<run-id>/response.json` (or any sibling bundle that follows the [zymtrace-anchored-query](/plugins/profile-and-optimize/skills/zymtrace-anchored-query/SKILL.md) layout) - derive `kernel_class_gpu_pct` from the GPU-side `event_kind = 'cuda'` query response, bucketed per the kernel-class regex appendix in that skill. Derive `cpu_spinpoll_pct` from the CPU-side `event_kind = 'on_cpu'` response by summing samples whose `py_file` matches `shm_broadcast.py` or `utils.py::sched_yield`. - If no paired profile bundle exists, emit `null` for both optional fields. The diff verb will skip them per the "Optional fields" rule above. The registry record stays valid. Emit `inference_perfbench_v1.json` to a scratch path, `sha256sum` it. #### Phase 2: register ```text mcp__profile_and_optimize__perf_baseline_record with: args: ["--family", "inference", "--measurement", "<model>-perfbench-v1", "--source", "<scratch-path-to-inference_perfbench_v1.json>", "--unit", "structured-json", "--notes", "<operator-supplied notes; e.g. GPQA + MMLU-Pro scores from inference-model-eval>", "--json"] ``` The verb writes the registry entry under `experiments/artifacts/perf-baselines/inference/<model>-perfbench-v1/<UTC-ts>/` (per the registry layout the upstream verb already implements). #### Phase 3: report Print: - Registry path. - Source SHA-256. - Headline metrics at concurrency 16 (the canonical operating point for routine comparison). - Cross-link to [`inference-perf-baseline-bridge`](/plugins/profile-and-optimize/skills/inference-perf-baseline-bridge/SKILL.md) `diff` for future regression checks. ### Mode 2: diff #### Phase 1: resolve baseline + current - `Read <baseline>/baseline.json` - confirm `family == "inference"` and `measurement == "<model>-perfbench-v1"`. - `Read <baseline>/source-snapshot.<ext>` - load the baseline's `inference_perfbench_v1.json`. - Parse the current bundle into the same schema shape (Phase 1 of record mode). #### Phase 2: per-metric diff with tolerances For every concurrency cell in `metrics.<concurrency>`, compute: - `delta = current - baseline` - `delta_pct = 100 * delta / baseline` (for percent metrics) - `delta_abs = current - baseline` (for absolute metrics: cache hit rate) - Per-metric verdict: GREEN if `|delta|` is within the tolerance for that metric (regression direction respected), RED otherwise. For `server_metrics_delta`, apply the tolerances at the absolute level: a `prefix_cache_hit_rate_post` drop of more than 2 percentage points is RED. A peak `gpu_cache_usage_perc` rise of more than 5 percentage points is RED. #### Phase 3: overall verdict