refresh-evidence

# Refresh and triage the evidence snapshot `data/localmaxxing-snapshot.js` is generated — never hand-edit it. It carries the model catalog (`models`) and the calibration corpus (`goldCases`, ≤240 rows). The engine's peer correction, the "nearest measured" ladder rung, the evidence workspace, and the landing-page quick start all read it. The neural.download lab rows live apart in `data/lab-evidence.json` (hand-edited; `npm test` regenerates `data/lab-evidence.js` and stamps its cache key) — see the `calibrate-engine` skill. ## Commands - `npm run refresh:localmaxxing` — rebuilds the snapshot from `https://www.localmaxxing.com/api` (`/models`, `/leaderboard`, paged) **and rewrites the `?v=` cache key in `index.html`**. It loads `engine.js` through the test harness for the prefill plausibility check, so the engine must parse. Commit the snapshot, `index.html`, and `dist/` (the SDK evidence bundle `npm test` rebuilds) together; committing only the data file leaves browsers on a cached snapshot and fails the cache-key integrity test. The workflow (`.github/workflows/refresh-localmaxxing.yml`) is manual-only, with publication off by default. It runs `npm test` + `npm run audit:gold` and uploads a review artifact; committing additionally requires the explicit `publish` input. Weekly refreshes were disabled after the September 8 review found snapshot-dependent numerical pins blocking new evidence. Those pins now use `tests/fixtures/calibration-2026-08-24.json`; do not refresh that fixture. The workflow additionally requires `audit:gold -- --strict-physical` (zero unresolved >5% roofline violations), while retaining existing statistical guards. It also blocks rows carrying `decodeMeasurementIssue`. The parser now stores zero for independent llama-bench tg and flags `-pg`/depth sweeps for original per-test JSON review; never silently remove these rows or reinterpret combined rates as isolated decode. Historical published rows await a guarded migration. - `npm run audit:gold` — distribution, per-runtime/hardware medians, roofline violations, worst rows. - `node scripts/fit-decode-constants.mjs --rows` — every row with depth, observed, predicted, physical. ## What a gold row is `normalizeGoldCase` in `scripts/refresh-localmaxxing.mjs` keeps a run only if it maps to a preset (`MODEL_PRESET_RULES`), a device template (`HARDWARE_RULES`), a runtime (`RUNTIME_KEYS`), and a quantization (`normalizeQuantization`); has a real engine invocation (not a "# Remote endpoint"); is batch 1 and non-speculative; is not a pruned/abliterated variant mapped onto a base preset; and is not a wall-clock capacity probe (prompt ≥ 32K, TTFT ≥ 10 s, no prefill rate). Its `prefillTokS` is dropped (decode kept) when the prompt-processing rate implies more than the device's dense tensor peak (`plausiblePrefillRate`: llama-server prompt-cache hits report 20–60k tok/s). `chooseGoldCases` then keeps up to 3 runs per (preset, hardware, devices, runtime, quant) signature and 16 per preset, most reproducible first; every hardware template keeps its best 4 rows before the rest fills by reproducibility, capped at 240 — so one new device with a few runs is calibratable immediately. Fields the engine depends on (keep them when editing the script): `contextLength` (configured window → KV *allocation*), `promptTokens`/`outputTokens` (decode *depth* = prompt + output/2, for llama-bench rows as well — their tg rates fall as 1/p with `-p`), `kvCacheDtype` (from the API flag or `-ctk`/`--kv-cache-dtype` in the command), `splitMode` (`-sm tensor|row` → tensor strategy), `deviceCount` (honours `-ts a/b` and `-tp N`), `peakVramGb` (proves residency for dense mixed-precision quants), `cpuMoeLayers` (`--n-cpu-moe N` / `-ncmoe N`, or `'all'` for `--cpu-moe` / `-ot exps=CPU`), `memoryGB` (the recorded pool; a smaller SKU than the template is honored), `backend`, `command`, `reproducibility`. ## Adding evidence for a new model or device 1. Add the preset / template first (see the `add-model` and `add-hardware` skills). 2. Add a `MODEL_PRESET_RULES` regex (HF id, base model resolves through `baseModel`) or a `HARDWARE_RULES` regex (hardware label). Specific patterns before general ones. 3. `npm run refresh:localmaxxing`, then check the new rows in `node scripts/fit-decode-constants.mjs --rows`. ## Triage checklist when the audit or the CI refresh fails 1. Read the failing guard: median drift, within-1.5×/2× share, optimized coverage, physical coverage, roofline violations (the integrity test prints the offending preset/hardware pairs). 2. For each offending row, fetch the raw run (`https://www.localmaxxing.com/api/leaderboard?limit=200&offset=N`, match by `id`) and read `notes`, `engineFlags.commandSnippet`, `promptTokens`, `ttftMs`, `tokSPrefill`, `peakVramGb`, `hardware.gpuCount`. 3. Decide which it is — and fix it at the source: - mislabeled quant/hardware/runtime → tighten `normalizeQuantization` / `HARDWARE_RULES`; - a different model than the preset (REAP, pruned, distill, "Ridge" quants) → own preset or exclude via the variant regex; - a measurement semantics the projection does not understand yet → extend `calculateGoldCaseProjection` (that is where prompt depth, KV dtype, split mode, and peak VRAM are applied) rather than special-casing a row; - genuine physics gap → `calibrate-engine` skill. 4. Never fix a guard by widening its threshold or by lowering a physical ceiling. Every violation so far was a data-semantics issue. 5. Re-run `npm test`, `npm run audit:gold`, commit the snapshot + `index.html` together. ## Data hygiene rules - Speculative/MTP rows are excluded from gold on purpose: the planner models speculation separately and labels it; mixing them in would inflate every baseline. `isSpeculative` checks the structured flags, `--spec-type` other than `none`, draft-model/`--speculative-config`/DFlash/DSpark/EAGLE/ngram spellings in the command, method phrases in `notes`, and "-mtp" checkpoints served by MLX-side servers (oMLX, mtplx). A decode rate above the physical roofline on a Mac is the usual tell that a speculative row slipped through. - Community "best" rates in the catalog (`bestTokS`) may be batched or speculative — display only, never calibration. - The snapshot must stay loadable without network and the app must degrade gracefully if it is missing (`LOCALMAXXING_DATA` fallback); tests run against the committed snapshot.