eval

# Eval — design and scaffold benchmarks that can't fool you Home base: `~/Projects/evals` (MACVB + standing bakeoffs + `evalcore` + scaffold). ## Why we run our own (operator framing, 2026-06-13) Evals here exist to **speed up, parallelize, and buy confidence that tokens aren't wasted — not to be stingy**. Two corollaries: - **Materiality threshold:** a routing/config difference is worth measuring — or even thinking about — at **≥1.5–2×** (tokens, wall, or quality). Below that, take the default and move; optimizing sub-1.5× deltas costs more in attention than it returns. (Matches the measured landscape: gated-execution effort tiers differ 0.3–0.65×, i.e. 1.5–3× — material; same-tier rewordings ~1.0–1.04× — noise.) - **Own-benchmark rule:** where literature exists, ADOPT its design (Phase 0); where it can't exist — frontier-local, per-release, per-rig properties like dispatch routing, effort knobs, harness/permission behavior, protocol compliance under failure — **your own eval is the only instrument there is**. Papers will always be a generation behind your rig. - **Ride real work:** the cheapest substrate is work that must happen anyway — dispatch the licensed-lane pieces as preregistered arms and the tokens pay twice (evals/dispatch_deletion_edges is the pattern: 3 production deletion edges shipped AND extended the routing table). **Sanity-check tier (below a full eval):** a ≤10-call probe with a written 1-line prediction, no scaffold, no DECISIONS row — for "does this lane even work / does this knob even move" questions. Keep the artifact (script + outputs) in the relevant eval dir or /tmp. The moment its result starts steering a default, it must retroactively pass the full gate (see Anti-patterns: "a quick probe that becomes a claim"). Methodology grounding: agent-infra `research/2026-06-11-eval-skill-and-evals-repo-improvements.md`, `research/benchmarking-science-2026.md`, `research/2026-05-31-eval-benchmark-methodology-delta.md`. **An eval is an instrument, not a script.** Most eval failures are instrument failures: softball cases that can't discriminate, judges that see candidate names, N too small for the claimed conclusion, golds nobody verified, results with no consumer. Every phase below exists because one of those happened here. ## Where does this benchmark live? (phenome / genomics / intel agents read this first) Two regimes — route by **cadence + entanglement + publishability**, the same rule BENCHMARKS.md already states. Both already exist in the wild; don't collapse them. | | Decision-grade bakeoff | Repo-coupled regression eval | |---|---|---| | **Question** | which model/engine/config? settle once | does my pipeline still produce correct output? | | **Cadence** | run once, record verdict | run every dev loop / in CI | | **Coupling** | low — reads prompts/data, owns no internals | high — tied to repo hooks/schema/pipeline | | **Home** | `~/Projects/evals` via `just new-eval` | the repo's own eval home (below) | | **Verdict** | DECISIONS.md row + production change | pass/fail gate in the suite | **Each repo already has its own eval home — use it; do NOT funnel everything to evals/.** The evals repo is specifically for *cross-cutting* model/engine/tooling-selection that spans repos or is about generic agent infra. Domain and strategy benchmarks stay home: | Repo | In-repo eval home | What goes there | |---|---|---| | **intel** | `~/Projects/intel-harness/` (backtest substrate; `bin/eval_proposed_rule.py`, walk-forward) | strategy/rule/thesis validation — has its own ruler-validity gate (`benchmark_validity()`). **NOT** evals/. | | **genomics** | `benchmarks/` + `benchmark_catalog.py` + GIAB baseline + QA-gate AUPRC/AUROC thresholds | variant-calling accuracy, scoring-tool gates, latency | | **phenome** | `eval/` (researcher-eval) + `tests/evals/epistemics/*.yaml` + `hook_mutation.py` | search/RAG quality, epistemic behavioral/calibration cases | | **any** | `~/Projects/evals` via `just new-eval` | "which extraction model / search engine / retrieval backend" — cross-repo, settle-once | So: `evals/extraction_bakeoff` benchmarks intel's *and* phenome's extract prompts from inside evals/ (cross-cutting model choice), but an intel strategy backtest goes to intel-harness and a genomics variant-accuracy check stays in `genomics/benchmarks`. The `/eval` **discipline** (pre-registration, power, blind judges, invariant-claim split) applies in all of these homes regardless of location. **Tooling — NO symlinks for code.** The house pattern for shared Python is a package in `substrate/packages/<pkg>` consumed via `path = "../substrate/packages/<pkg>", editable = true` (how corpus-core / corpus-testing reach phenome+genomics). Symlinks are only for data dirs and the AGENTS.md/GEMINI.md→CLAUDE.md doc mirrors. - **`evalcore` lives in `~/Projects/evals/evalcore` and is evals-local.** evals/ is `package = false`; its bakeoffs import `evalcore` from the repo root and get the scaffold + prereg guard for free. Pure-stdlib, zero-dep. It was promoted to `substrate/packages/evalcore` on 2026-06-13 and returned on 2026-07-14 (evals@4ab4bde) when its only second consumer, phenome, retired; the substrate copy was deleted 2026-09-23. Key imports: - `from evalcore.run import run_eval` — the guarded loop: leak-checked trials, a transport pre-flight that aborts a dead arm before the sweep, one atomic `{run_id}.trials.jsonl` - `from evalcore.judge import dispatch, assert_blind, lint_not_leading, lint_stakes_neutral, cyclic_assignment` - `from evalcore.leakguard import assert_no_gold_leak` — call before EVERY system-under-test dispatch - `from evalcore.stats import wilson_ci, mcnemar_exact, cohen_kappa, holm_correction, point_biserial, benjamini_hochberg` - **An in-repo eval elsewhere that needs these primitives is the second consumer** — re-promote evalcore into `substrate/packages/` under the proven-common test (agent-infra `decisions/2026-06-09-shared-extraction-proven-common-test.md`) and point both repos at it. Until then no other repo carries the dep. Still: NO symlinks, NO copying evalcore's code into a repo. ## Phase 0 — Dedup (before designing anything) ```bash cat ~/Projects/evals/BENCHMARKS.md ~/Projects/evals/DECISIONS.md # settled questions grep -il "<topic>" ~/Projects/agent-infra/.claude/rules/vetoed-decisions.md ``` If the question is settled (extraction models, SaC routing, retrieval backend, judge panels...), the answer is the DECISIONS.md row — don't re-run it. New evidence that a verdict is stale → re-open via a new run, never by editing the old verdict. **Also check the external prior art — for the DESIGN, even when the rates are stale.** A 2-minute search (Exa/Perplexity for the named bias/benchmark + arXiv) tells you (a) whether a benchmark already exists (don't reinvent — `JudgeBiasBench`, the SPB PIR/Null-PIR framework, etc. existed and we nearly rebuilt them) and (b) **the controls the field already knows you need**. "Measure the live model, papers lag the frontier" (frontier-timeliness rule) licenses re-measuring the *rate* on current models — it does NOT license skipping the literature's *method*. Papers lag on rates but LEAD on confounds: the controlled design (length-ratio control, truncation control, quality-matched neighborhoods) is scale-independent and transfers. Skipping it is how you repeat a confound the field solved two years ago (see the verbosity egg in Anti-patterns). **For a structured fitness verdict on a candidate external benchmark, dispatch the `benchmark-rater` agent** (global; preloads this skill + `model-guide`). It grounds the rate-vs-method call in real sources (the paper, the scoring code, training-set membership) and returns **USE-AS-IS / ADAPT-DESIGN-ONLY / REJECT** plus a **contamination–durability** sub-verdict (DURABLE / ROTS / NEVER — does it rot as a standing instrument re-run per model release?). Use it before adopting a third-party benchmark *and* before greenlighting a from-scratch build — its most common output, ADAPT-DESIGN-ONLY with a precise BORROW list, is what stops both a needless rebuild and a naive adoption of a rotting benchmark. **Promise-to-release is not released:** grade DESIGN from the paper if useful, but RESULT stays non-auditable until the task/grader/record-level ledger and exact run manifest are actually available. A future-tense repository link earns no reproducibility credit. ## Phase 1 — Design (the questions that decide if the eval should exist) Answer in writing (they become PREREGISTRATION.md fields): 1. **Construct** — what ability/property, operationalized how? One sentence. *Estimand before leaderboard:* declare whether the unit is a **model**, a **harness**, or the deployable **model×harness configuration**. A configuration leaderboard is valid for exact-system routing; it does not identify a model effect. Model claims need a common harness or a connected factorial grid with model, harness, and interaction effects reported separately. 2. **Decision + consumer** — which production default / routing rule does the verdict change, and where is that recorded (DECISIONS.md row)? **No consumer → don't build.** *Criterion validity (the NASA-recruiter test):* name the real-world outcome the score is a proxy for — the way a selection test is only worth running if it predicts *job performance*, not test-taking skill (Schmidt-Hunter selection-validity literature). "Extraction F1" matters only insofar as it predicts pipeline usefulness; a benchmark you never check against the downstream outcome is a vanity metric, however internally clean. Where the criterion is measurable, validate against it; where it isn't yet (partial-verifier regime), say so and treat the score as a *bounded* proxy, not the target. (This is consumption-over-autonomy with a name.) 3. **Verifier regime** — deterministic ground truth (substring, recall@k, exact answer) or judged? Deterministic PRIMARY decides; judges corroborate. A model-as-judge proxy does not make taste work verifiable (constitution: bad eval is worse than none). *Spec↔oracle alignment:* a deterministic oracle must enforce exactly what the task states — no more (overly-strict → false-fails correct solutions; a pass then evidences contamination, not skill) and no less (low-coverage → false-passes incomplete ones). Executable ≠ valid; see the mined-task audit under "Grading an EXTERNAL / vendor benchmark". *Input availability is part of the prompt:* for file/tool agents, the presence or absence of one special file can reveal the intended method. Stage a realistic **superset** of plausible inputs and run a cue-only baseline. Pre-register the distractor policy and match production clutter; excessive or semantically alien distractors change the construct into triage/context management. *For retrieval/ranking:* is there exactly ONE relevant item per query, or can the corpus hold co-relevant siblings? One-gold recall@k is valid only in the former; topically-dense corpora need graded multi-doc relevance, or recall@k scores label noise (see Phase 3). *For free-text "did they identify X?" grading:* substring/anchor matching is paraphrase-brittle and biased against the arm that words it differently (critique_replay: a scored "universal MISS" was a universal HIT — the anchor caught one arm's phrasing). INVERT the default there — a blind dual-family judge (**cite-required**: a DETECTED with no quotable span is NOT_FOUND) is PRIMARY; substring + cross-arm convergence run as deterministic BACKSTOPS that FLAG (never silently override) disagreements for human resolution (evals ADR 0005). Three judge-transport gotchas that contaminated a real run *before* its verdict — all caught by reading traces, not buckets: - **INLINE the payload into the judge prompt; do NOT `-f`-attach it.** Under grading framing the judge silently mis-reads an attached file ("no reviewer findings were provided") though it IS delivered — reproduced deterministically. (Fix belongs in `evalcore.judge.dispatch`.) - **A "no findings" / JUDGE_ERROR verdict is TRANSPORT, not a miss** — read the rationale; SMOKE one packet before the batch (it caught two transport bugs before any spend; separate transport from capability). - **Judge competence is ASYMMETRIC — validate each judge against ground truth before trusting a panel.** Cross-family-PRIMARY routing assumes equal competence; a weaker judge (gemini-3.5-flash, forced temp=1.0) under-counts the arms it is primary for. κ + human-resolve-disagreements is load-bearing, not optional. 4. **Criterion over pipeline** — rubric/criterion design explains ~9× more judge-reliability variance than scoring architecture. Spend the hour on the rubric, not on a fancier panel. 5. **Contamination plan** — per-item provenance: `authored-fresh | post-cutoff | public-lifted`. Public items contaminate candidates AND judge memory. `authored-fresh` means the *exact text* is novel — NOT that the topic is; general-science/medical content is in every model's training data even when your composition isn't, so don't over-claim contamination-freeness. - **Valid gold for a claim/verification eval is AFFIRMATIVE, never absence.** A refuse/abstain/HOLD gold must come from a confirmed contradicting source, retraction, or supersession — NOT from "not in the store / NEI / no evidence found" (closed-world; inverts rankings). Every external claim-verification benchmark's NEI label is absence — do NOT lift it. (Canonical: phenome `reference_kb_grounded_eval_defeater_invariant` + ADR 0008; the SciFact/JudgeBench/COVID-Fact audits.) - **The strongest uncontaminated gold is a POST-CUTOFF VERDICT FLIP — harvestable, not just a tag.** A claim whose authoritative verdict flipped *after* the model's cutoff T is contamination-free *by construction* (the model can only have memorised the pre-flip verdict → it regurgitates the now-wrong belief) AND an affirmative defeater (anchored to a dated authority). That combination is the one thing no borrowed benchmark supplies. Harvest LIVE (recall forbidden — past your cutoff too) from dated-authority feeds filtered to `change_date > T`: Retraction Watch, the FDA DSC table, ClinVar delta, the CPIC `cpic-data` git log. Cite the primary dated record, not a blog's crawl date (mirrors re-surface old posts with fresh dates). This is intel-harness point-in-time with **T = the training cutoff**. Method + first harvest: `~/Projects/evals/docs/post-cutoff-flips/`. - **Enforce the flip-gold with the SEAL, not trust — and split judgment from retrieval.** Flip-gold is clean only if the SUT can't search its way to the post-T answer mid-eval, so pair it with `allow_internet=false` (the DeepSWE seal below): the airgap turns "we assume it didn't cheat" into "it provably couldn't." This also dissolves the "but it doesn't test web-browsing" objection by SCOPE: a *judgment*-over-a-staged-claim eval (adjudicate promote/hold/reject given an evidence **packet**) is packetizable + airgappable; *retrieval* ("find the right materials") is a SEPARATE capability where PIT-constraining a LIVE search is unsolved (URLs mutate in place at stable addresses, date metadata lies, engines have no as-of-T mode — intel-harness's unsolved web-PIT problem). Claim-verification ESCAPES that trap precisely because a claim+evidence bundle is a static object you curate as-of-T once; trading can't, its task is integrating an unbounded flowing stream. **Curating the packet enforces the PIT boundary by construction — don't try to PIT a live search.** If you must test retrieval, do it against a FROZEN snapshot (Wayback/Common Crawl at T) and state the coverage ceiling; don't pretend live fetch is as-of-T. - **Same-family confound — at EVERY model touchpoint** (generator / judge / reranker-fusion): any model that helps produce or score the eval's relevance signal must share no lab/family with any candidate. ≥2 neutral-family generators and judges (blinding does NOT protect — the bias rides the content of the grades); a candidate-family reranker informs, never gates. Measured incidents + mechanics: [references/anti-pattern-evidence.md](references/anti-pattern-evidence.md) §Phase-1. - **Cheap pre-screen before building an embedder bakeoff at all:** embed the corpus with both candidates, run a handful of queries through the *production* stack, measure top-10 overlap. **>~0.85 ⇒ the swap is immaterial — don't build the full graded eval.** (v2 measured 0.89: through hybrid+rerank the two embedders returned ~9 of the same 10 docs. The full eval confirmed HOLD, but the overlap screen predicts "switching changes ~1 in 10" in minutes — run it at the §0 gate.) - **Frontier judge bias (measured 2026-06):** position/order bias is SOLVED on frontier judges — stop spending order-swaps/position-debias there. Same-lab judges share a style signature (a same-lab panel ≈ 1 effective vote; Bradley-Terry SE on it is false precision) → use cross-lab panels, read disagreement as signal. Our "verbosity bias" probe was RETRACTED (length-ratio artifact). Rates, arXiv refs, re-run controls: references/anti-pattern-evidence.md §Phase-1. - **Judge NOISE BUDGET — a single-trial judged number is PRELIMINARY** (distinct from bias): ~13.6% mean single-trial flip rate at frontier; for a decision repeat the judging (≥3) or PPI/PRECISE-correct; a same-lab multi-judge panel is NOT independent votes. Numbers + refs: references/anti-pattern-evidence.md §Phase-1. 6. **Invariant ambition** — what mechanism-level claim could this eval produce that survives a config swap? If only a local verdict is possible, fine — say so up front. ## Phase 2 — Scaffold ```bash cd ~/Projects/evals && just new-eval <slug> just power <N> # paste output into PREREGISTRATION.md; declare SCREENING|CONFIRMATORY git add <slug>/PREREGISTRATION.md && git commit # prereg FIRST — the guard enforces this ``` Golds are **platinum**: mechanically checkable or human-verified. Never audit golds by asking an LLM to re-check them (auditors re-solve and trust themselves: wrong-reference detection 68%→9% at scale). **Every gate band comes from this eval's own null, at its actual N**, before any arm is scored: negative-control ranges from ≥200 label shuffles (or the analytic SE), margins checked against the bootstrap SE, and the whole control set's chance false-alarm rate stated (≤5%, adjusted for the number of controls). Never copy a band from an earlier, larger run. Genomics SpliceAI2 follow-up (2026-10-26): a 0.45–0.55 shuffle band inherited from a run with ~400× more pairs sat ±1.65 SD from the null mean at 624 pairs, so about a third of runs failed it on chance, and a +0.053 AUROC win came back INCONCLUSIVE. ## Phase 3 — Discrimination probe (before budget) `uv run python3 <slug>/run.py --probe` on ≤10 items. Required: a trivial-pass baseline score AND one case that separates candidates. Both flat → fix cases, not N. (Evidence: File-Search-vs-emb "parity" was title-matching softballs; discriminating rerun: 8/10 vs 4/10.) **The probe is NECESSARY, NOT SUFFICIENT — eyeball misses + check gold validity.** A low trivial-baseline rules out lexical leakage; it does not rule out label noise or over-obfuscation. Before trusting ANY recall/nDCG/accuracy number: - **Eyeball ≥10 actual misses** — read the query, the gold, and *what scored above the gold*. If the higher-ranked items are plausibly co-relevant/correct, your label is wrong: you are scoring **label noise as model failure** (and can penalize the BETTER system, which clusters co-relevant items). Cheapest validity check there is; the most-skipped. Watching aggregate ranks is NOT this — look at *what beat the gold*. - **Single-gold → graded relevance on topically-dense corpora.** One-gold recall@k is valid only if exactly one item is relevant per query. If the corpus has sibling/overlapping items (multiple docs on a topic, near-dupes), collect a relevance *set* (LLM-judge the top-k union + human spot-check), score graded nDCG/MAP. A verdict that flips across analyst choices (generator, pool, window) is the tell that label noise ≫ signal. - **Don't over-obfuscate to beat the baseline.** Aggressive keyword-stripping can push queries into unrealistic riddles that test abstraction, not retrieval — match the real query distribution. (Evidence: bio_embedding_bakeoff 2026-06-11 — kw-baseline 0/12 looked rigorous, but **23/48 queries had the gold outranked by co-relevant sibling memos**; the single-gold recall@5 verdict was *inconclusive*, surfaced only by eyeballing misses post-hoc — `audit_label_noise.py`.) **This generalizes beyond retrieval — read the TRACES, not the buckets, for ANY eval, and ESPECIALLY on a clean/perfect score.** Aggregate buckets are a proxy; a verdict on unread traces verifies the arithmetic, not the construct. Persist traces by default (a harness that discards them is unauditable).