inference-value-ledger

# inference-value-ledger One command to render the always-current leadership value-prop view. ## Run it ```bash # print to stdout: perftunereport value_view # write a doc: perftunereport value_view --out /path/to/VALUE-LEDGER.generated.md # JSON envelope (finding + flag counts, e.g. for CI): perftunereport value_view --json ``` `perftunereport` ships with the profile_and_optimize server. If it is not on PATH, put the server on PYTHONPATH and invoke it as a module. ## How it works - **Curated source-of-truth:** `perf-tune-report/configs/value-findings.yaml` - one entry per finding (`id`, `title`, `lifecycle`, `baseline`, `win`, `hardware`, `deploy_readiness`, `campaign_ids`, **`next_lever`** [+ optional `next_value`]). Edit this when a finding's status changes or a new finding lands. This is the ONLY hand-maintained part. - **`perftunereport value_view`** joins it with the LIVE perf-lake campaigns: each campaign's `report_status.json` (`sol_rigor` / `dcgm_grounded`) + `verdict.json` (`tier` / `baseline_named`), and renders the grouped table **plus the ranked GRIND FRONTIER**. Win numbers are human-verified in the registry. The live columns + flags are read at render time so the ledger never silently drifts from the lake. - **Flags surface** when a campaign is missing locally (S3-only / unpublished), ungrounded (`sol_rigor` none/L1), its baseline is not named, or **a finding has no `next_lever`** - fix by publishing/tagging the campaign or naming the next lever. ## Keep it honest Lifecycle is evidence-state, not aspiration: a finding is `done` only when its campaign is published + grounded, `in_progress` while revalidating, `not_done` for research, `closed_negative` for banked dead-ends (value = prevented spend). Never mix B200 (TP=8) and GB300 (TP=4) without the hardware column. Every row names its baseline (the DRAFT-vs-VERDICT "fair baseline" rule). **Code-under-test provenance match:** a finding's `source_refs` `delivery`+`commit` MUST match the cited `campaign_ids`' provenance -- citing an `overlay`/offline-prepped campaign as the benefit of an `infr-patch` is a cross-tier DRAFT defect (even when the kernels match). `value_view` flags this via `provenance_match_problems()` (delivery/ commit mismatch between `source_refs` and a cited campaign). Typical failure: citing an overlay-deploy champion's number as a source patch's benefit -- always re-measure on the patch's own deploy. ## Full-context reporting (no bare numbers) Per the methodology canon "Every performance number carries its full context (no bare numbers)" (`docs/METHODOLOGY.md`, "Full-context reporting"): every number this skill emits (throughput, latency, TPOT/ITL, BW, %SoL, speedup, efficiency, goodput, acceptance rate, scaling efficiency, thermal/failure rate - whatever it reports) MUST carry its full measurement-context descriptor, and every comparison MUST be matched on it. A bare number is a defect - it cannot set a default, ship a config, or appear in a report. - **Identity:** model (+HF path), hardware (exact ceiling token `GB300`/`B200`), quant, kv-cache dtype. - **Parallelism:** TP, DP (replicas), PP, EP, parallel_strategy. - **Serving cfg:** max-num-seqs, max-num-batched-tokens, gpu-memory-utilization, max-model-len, cudagraph_mode/enforce_eager, async_scheduling, prefix-caching. - **Workload:** dataset, ISL/OSL (or mean in/out tokens), concurrency, num-prompts. - **Regime:** warm vs cold. Latency vs throughput tier. - **Stack:** image/vllm commit, bench backend, serving engine. - **Grounding:** `%SoL` (+ ceiling key from `configs/sol-ceilings.yaml` - never inline a peak), sol_rigor (L1-L4), trials n (mean±std), same-node, baseline named. (If the metric is not roofline-bound - e.g. accuracy/acceptance - omit `%SoL` but keep the rest of the descriptor.) - **Per-number exact shape (no smoothing):** when reporting more than one number, keep EACH with its own exact shape (ISL/OSL, concurrency, dataset, regime) - never normalize a set to one uniform descriptor that hides per-point variation (e.g. `c=1 @ ISL1024/OSL256` + `c=64 @ ISL4096/OSL512`, NOT one shared "random"). ## Asset validation (review + FAIL LOUD) Every asset this skill emits (value-prop ledger / GRIND FRONTIER view / table) is held to the asset-validation canon (`docs/METHODOLOGY.md`, "Asset validation"): the generator **FAILS LOUDLY** on missing/bad data (a finding whose backing campaign is missing or ungrounded, `unknown`/null where a value is required) -> flag it loudly, never silently drop or fabricate a row, and the agent **REVIEWS** the ledger for human-sense + 100% accuracy (every win row is campaign-grounded with live sol_rigor + verdict tier, no orphaned/ungrounded claim presented as a win) and **rebuilds** it if wrong -- never ships a wrong/confusing ledger with a caveat. ## PR value proposition (WHY + benefit + applicability) This skill IS the value-prop ledger -- so when one of its findings graduates into a PR (a downstream patch PR or any upstream/shareable PR), that PR MUST carry the same value framing, not just the metric: the **WHY** (blocker removed / capability unlocked), the **measured benefit** (full descriptor + `%SoL`/`sol_rigor` + a resolving perf-lake / evidence link. Never a hand-waved "no perf penalty"), and an **applicability matrix** -- model architecture (sparse/MoE vs dense), size, reasoning-vs-not, quant, parallelism (incl. any TP>1-specific behavior), workload, and concurrency regime. State what it does NOT apply to. **Keep it tight (no AI-slop):** lead with SoL%/roofline% and matched numbers. Cut hedging, redundancy, and decorative charts (infra PRs don't embed images). De-slop checklist: no em-dashes (#1 AI-slop tell), minimal bold, plain punctuation, inline code out of narrow table cells. Source: `docs/METHODOLOGY.md` "Value proposition" + "De-slop". ## Next lever / BREAKTHROUGH (Grind Mandate) If this skill emits a measured result, its output MUST end by naming the **next perf lever**, its **expected unlock** (direction + rough magnitude), and the **gate** that proves/refutes it, per the Grind Mandate (`docs/METHODOLOGY.md`, "Always be grinding"). A measured win is the new floor, not the finish -- so **do everything we can to find the next BREAKTHROUGH**: the highest-EV unlock toward Speed-of-Light (a new champion / kernel / router / quant / parallelism / spec-decode win, or an unblocked stack), not just the next micro-lever. Rank the candidate breakthrough levers by value x cost (the GRIND FRONTIER, `perftunereport value_view`), pursue the top, bank the rest with evidence. Record WHY a refuted lever loses, update the standing frontier in the active bundle's `HANDOFF.md`. Never conclude "exhausted/optimal/done" without an explicit next-lever frontier (an empty frontier AND a documented SoL wall only). Delete this section ONLY if the skill produces no measurements.