benchmark

# Benchmark Handle $ARGUMENTS. For `review <scope-or-plan>`, read the full [performance review method](./references/performance-review.md). This mode inspects the plan and proof requirements without starting a measurement or implementation loop. Continue below when the request calls for measurement, causal diagnosis or a repair. Read [methodology.md](./references/methodology.md) completely before creating a plan, running a benchmark, or changing runtime code. The Perf issue playbook (`.agents/playbooks/perf-issue.md`) runs its measurement and repair lifecycle: intake, the plan, ordered execution, the conclusive cause gate, the durable fix decision, fix-rerun-resume and the close. This skill holds the lanes, harness, cohorts, budgets, comparison law, correctness guards and the pre-acceptance probe. Read that playbook completely before creating a plan, running a benchmark or changing runtime code. Before any benchmark row or result is logged `verified`, in any mode, run `pstack:benchmark-checklist` on it. ## Public Surface - `$benchmark review <scope-or-plan>`: review cohorts, budgets, interaction, memory, native-behavior and production proof requirements without execution. - `$benchmark [scope]`: discover and run all applicable lanes in diagnostic order. A scope narrows fixtures and owners, not the default lane inventory. - `$benchmark only <lane-or-target>`: explicitly narrow the run. Keep every default lane in the plan and mark unrelated rows `N/A: only - explicitly narrowed`. A whole-project inventory still measures existing specialized lanes such as pagination. Large documents, virtualization and performance scopes do not imply pagination work; change pagination only on an explicit pagination request. ## Pre-Acceptance Architecture Probe Architecture and API owners invoke Benchmark before accepting a target when it adds, retains, or changes a runtime layer, cache, index, projection, store, subscription, scheduler, geometry owner, repeated-unit fan-out, or other hot work. This is an embedded worker probe in the active architecture plan, not a new public mode and not a second full Benchmark plan. Before the architecture owner may lock that target: 1. Identify the user operation, current owner, proposed owner, independent scale variables, repeated units, and applicable normal, large, stress, and pathological cohorts. Scale variables include the feature's own state, such as blocks that hold a suggestion, not only the document's size. The test that counts the operation's work in `pnpm check` sweeps each one, from the first operation on. 2. Freeze the absolute/relative budget and noise rule before reading the target result. Reuse an honest existing budget when one exists. Run each budget line on the baseline first. A line the baseline already fails or nears measures existing cost, so restate it before freezing. Set a time budget by how often the operation runs, so a per-keystroke path gets a tight budget and a once-per-gesture choice does not. Size each line relative to the baseline and above the timer resolution, and run one planted-slowdown control that the line must reject; an absolute tolerance larger than the measured quantity cannot fail. jsdom timing cannot measure a style or layout read, so it never decides whether one belongs on the path. 3. Capture a comparable current-owner baseline. If the proposed path is not executable, build the smallest disposable prototype that can falsify its claimed owner and scaling law; do not start production implementation. Time the baseline only on cohorts where it does the candidate's work; a baseline that refuses or reads the input literally is a control, not a comparison. Run a baseline at a commit in a worktree from `node tooling/scripts/proof-worktree.mjs`, as the Base worktree section of `verify`'s command recipes says, and copy candidate files into a second worktree only through its `--with`. 4. Run matched baseline/target cohorts and record source identity, fixture, action, environment, sampling/noise, deterministic work indicators, timing, and the correctness/native guard. The probe writes every cohort's measurements to an attempt-indexed log in the run directory before it asserts any line, then asserts every frozen line itself and fails when a budget input is missing. A paired wall-clock ratio line runs through `node .agents/rules/benchmark/scripts/paired-timing.mjs`. The runner refuses a config unless every frozen cohort has a planted arm, which names the ratio its slowdown plants, and a same-code arm. It judges each cohort apart. A cohort's noise is the larger of its same-code gap and the furthest any pair of the candidate or a same-code arm reads from that arm's median, because a same-code arm divides by the same base runs as the candidate and cannot see base's own noise. A cohort is inconclusive while a planted arm exceeds the budget by no more than that noise, while it reads further from the ratio it plants than that ratio's distance to the budget plus the noise, or while the noise reaches the candidate's distance to the budget. A conclusive cohort therefore has every candidate pair on one side of the budget. A conclusive miss on any cohort fails the run. Only a measured run that passes every line rewrites a tracked receipt. It also runs once on a known-bad input, which must fail with the probe's own assertion text, as Harness Repair asks of a gate. Each cohort reads its mode and workload back from the running page, such as the rendering mode and the block and page counts, and fails when they differ from what its name claims, so an ignored or renamed parameter cannot measure a default. Every arm, the known-bad one included, shows it did the work its name claims; a layout-thrash arm that writes an unchanged value dirties nothing. A Vitest probe that prints its numbers runs with `--reporter=verbose --silent=false`, because Vitest's agent reporter hides them behind a bare pass. 5. Accept the target only when the result meets the frozen contract and the evidence isolates the claimed owner. An inconclusive result leaves the decision `defer` or `gate` with the next probe named. A `gate` writes that probe and its frozen budget into the step that runs it, in whichever plan owns that step, with the rule that a miss sends the target back to `architect`. Keeping or dismissing code on a cost argument needs a matched run with that code removed. An asymptotic table, review score, profiler suspicion, future benchmark plan, or "measure during implementation" note cannot satisfy this probe. Record N/A only when live source proves the change is type-only or cannot change repeated or hot runtime work. The probe never goes to a session card, because the target's acceptance waits on it. It narrows its cohorts or rounds to fit the block's one-hour Time budget, and an inconclusive narrowed probe leaves the decision `defer` or `gate`, as step 5 says. After implementation, rerun the same contract on the final production path and source identity, then run the correctness guard. While a fix is still changing, check it with one quick pass of the affected rows; the full interleaved protocol runs once, on the final tree, and again only when a later change touches the measured path or a result misses its budget. Each run fits the block's one-hour Time budget. A protocol that cannot fit, or that needs a quieter host than the run has, goes to a session card as that rule says. Its lines stay `gate`s that the run's close does not wait on. In a Perf issue run, whose validator does not yet accept a deferred measurement, the plan stays partial instead, as the close rule says. The lead sets that `Status:` before it offers the task, so the task waits on no `run:`, which only a done plan frees. A disposable prototype proves the design decision only; it never proves the shipped implementation. When that rerun conclusively misses a frozen line under its noise rule, the target goes back to `architect` at once, as the Perf issue playbook's Pivot section says; an inconclusive line stays a `gate`. ## Comparison Law - Current Plate versus main Plate is product truth. Use isolated source-built hosts, exact refs or dirty fingerprints, and the same route/fixture/action. - Plate versus Plite is layer attribution. Use the same current source, document, renderer intent, rendering component and options, and action; report wrapper/plugin tax without calling Plite a Slate baseline. - Plite versus Slate is substrate evidence. Pin and record the exact local Slate commit; a mutable `../slate` path is not a baseline identity. - Product and engine comparisons answer different questions. Run both when both are applicable; never substitute a minimal engine win for a real app route. - Mount and trusted editing are primary. Heap/DOM are supporting guards unless memory, DOM growth, or a degraded mode is a plausible cause. - When the measured diff also holds an unrelated performance fix, measure the feature against a baseline with that fix applied, and report the fix's effect separately, so its speedup cannot hide the feature's cost. - Before attributing a metric shift or calling a result the first of its kind, look up earlier values in durable history (`node tooling/scripts/review-ledger.mjs lookup <scope>`, `docs/plans`), not only in scratch artifacts. ## Correctness And Native Behavior Before keeping a fix, answer the matching rows of `benchmark review`'s Performance Questions for cohorts, interaction metrics, repeated units, readiness, degradation, and native behavior. Use real keyboard/browser input for user-facing editing claims. Programmatic transforms may isolate runtime cost but stay labeled as proxies. If no correctness oracle exists, pause the perf claim and create the smallest one through pstack's Bug fix playbook. Do not bury missing behavior proof inside a benchmark artifact. A candidate's correctness oracle never comes from the build it judges, so a performance-only candidate's output must match the baseline's. A candidate that changes output on purpose asserts each side's expected output shape for every cohort before timing, because a fast refusal or a literal fallback also reports success. ## Harness Repair Repair the benchmark before product code when source identity, host freshness, fixture parity, sample count, percentile math, metric aggregation, browser action, correctness guard, or artifact provenance is wrong. A failing producer guard is a product signal until a minimal repro shows the guard itself is wrong. A benchmark row measures only what its name says: helper reads such as full-snapshot materialization stay out of rows that time writes, reads or normalization, and a row that measures snapshots is named as one. Two identical builds prove only that an evaluator raises no false alarm. Prove a new or repaired evaluator end to end with at least one known-bad candidate; an identical-build result close to a guard means that guard sits inside the noise. A change to a shared benchmark summary or stats helper proves itself on one current core artifact and one current React artifact, read from the emitted JSON, before any threshold uses the new field. A mutant or candidate measured through a build counts only after the run shows its bytes reached that build, such as through a changed build fingerprint. A `*_SKIP_BUILD=1` run builds nothing and checks nothing for staleness. Before a Plite browser lane, run `node apps/plite/scripts/build-app-if-stale.mjs`, which rebuilds the example app; `build-browser-if-stale.mjs` checks only `@platejs/test`. Before a core lane, rebuild the package `dist` it imports. `benchmarks/editor` is a standalone npm package with its own lockfile. Its Evidence Kit CLI exists only after `pnpm bench:editor:install`, and the root `pnpm bench:editor:*` scripts wrap its commands. Do not create a second target registry. Discover current targets from `benchmarks/targets/slate-v2.json`, package scripts, app runners, and live test source. Add or repair a target only when the measured decision has no honest owner. ## Verification For Benchmark skill or methodology changes: ```bash node --test \ .agents/rules/benchmark/scripts/benchmark-contract.test.mjs \ .agents/skills/benchmark/scripts/benchmark-contract.test.mjs node .agents/rules/plate-next/scripts/sync-resources.mjs --check ```