trustworthy-cuj-scoring

name

trustworthy-cuj-scoring

description

Use when building or running an agent eval, CUJ score, benchmark, or automated-improvement loop — and especially when a whole lane fails uniformly, a score looks too good or too bad to be true, you're about to trust a new grader/oracle/judge, or a scored harness invokes pnpm / npm / tsc / any .cmd subprocess on Windows. Symptoms: "the agent can't do X" reads as a capability gap, exit-127 / exit 127, a lane scores 0% or 100%, judge and oracle scores blended into one number, write-only-at-end eval crashed and lost work, ≥2 different failure modes on one harness, hill-climbing against a contaminated signal. Keywords: oracle, grader, LLM-as-judge, position bias, Cohen kappa, sealed holdout, decontamination, SNR, MDE, noise floor, checkpoint, status.jsonl, broken oracle, reward hacking, blind relaunch.

Trustworthy CUJ Scoring

Overview

A broken oracle reads as a capability gap. Before you trust any CUJ/agent score, the deterministic floor must be validated bidirectionally and the judge must be quarantined — a grader that silently passes wrong answers (or fails right ones) makes the harness optimize against a phantom, and any loop built on it hill-climbs in the wrong direction.

Build the score as a cascade: deterministic oracles are the gate; the LLM-judge is a separate, last-resort layer for what no parser can express (semantic, taste, reward-hacking).

Full method, the LLM-judge statistics, and the 31 grounding sources live in methodology.md — read it when building a harness, designing a judge protocol, or defending a reported number.

When to Use

• Building or running an agent eval / CUJ score / agentic benchmark / automated-improvement loop.
• A whole lane fails (or passes) uniformly — check the grader first, not the model.
• A score looks too good or too bad to be true.
• You're about to trust a new oracle, grader, or judge.
• On Windows, the harness shells out to pnpm / npm / tsc / any .cmd wrapper.

Not for: rating a web surface (use rate-fix-loop); multi-benchmark eval-driver fault isolation (a separate suite-resilience concern); the broader CUJ→cluster→fix loop (a separate harness-evolution concern).

The gates (checklist — run in order)

1. Bidirectional oracle validation BEFORE any scored batch. For every grader, run one known-correct input (MUST pass) AND one known-wrong / empty input (MUST fail). If either direction is wrong, the grader is broken — stop. (WebArena passed empty replies; tau-bench's do-nothing succeeded on 38% of tasks.) 1b. Output-format / answer-equivalence validation. Bidirectional validity proves the oracle DISCRIMINATES (right vs wrong); it does NOT prove it accepts every VALID answer. Also run a known-correct answer whose format differs from the reference (same numeric value at different precision, JSON keys reordered, int vs str, currency/percent symbols, thousands separators) — it MUST pass. If it fails, NORMALIZE before exact-match (strip/case/round/sort-keys/set-compare) OR pin the exact format in the task spec. Right-but- differently-formatted answers are false GAPs that DEFLATE the category — a 25–52 pp false- negative gap between strict and normalized scoring is measured in the wild (Ganapaneni 2026; CFMatch 2024). Receipts: a numeric-precision case + a JSON int-vs-str case ( key-order) both scored GAP while trivially correct.
2. Deterministic floor, THEN a quarantined judge. Code graders (exit-code, regex, schema, AST, unit test, exact-match) are the gate. The judge runs only after the floor passes, only for what no parser can express, and its score is a separate non-blended track — never averaged into the deterministic headline. (Reward-hacking detection is the one thing only the judge can do: a test can't see a deleted assertion.)
3. Per-task append-only checkpoint. Write+flush one status.jsonl record per completed task; write run_config.json with the full task-ID list at run start; resume re-runs only errored+never-started. Never write-only-at-end.
4. Pre-state the SNR / MDE / noise band before the batch (N, baseline, minimum detectable effect, confidence). A result outside 2σ triggers a noise investigation, not a capability conclusion. Don't use the CLT under a few hundred datapoints.
5. Sealed holdout + pre-registered gates. Lock the pass/fail thresholds before any model is scored; test gold answers stay private. Treat a "clean" contamination scan as weak evidence — RL/GRPO training conceals contamination signals.
6. ≥2 distinct failure modes = rework, NOT relaunch. Diagnose the root cause (subprocess, OS shell behavior, env leak) before re-running. Verify liveness before relaunch; ONE owner per run.
7. Windows .cmd smoke-test. pnpm.cmd / tsc.cmd via list-argv subprocess returns exit-127 without shutil.which() or shell=True — it falsely fails the whole lane. Smoke- test the invocation on the target OS with a known-good pair before the batch.

Quick Reference — run discipline

Phase	Do
Pre-run	oracle self-test (good PASS + bad FAIL) · pre-register gates · state MDE/SNR · smoke-test .cmd on target OS · init run_config.json + status.jsonl
During	append+flush per-task checkpoint · asyncio.wait_for(timeout=N) per task · ONE owner
Post-run	judge as separate track (N≥3, AB+BA, Cohen's κ) · check vs pre-stated band · cluster failures · match fix-lever to cluster cardinality · retest before/after
Never	blind-relaunch after ≥2 failure modes · blend judge into the gate · trust a "clean" decon scan · single-trial judge on a high-stakes gate · CLT on small-N

Common Mistakes

• Reading a uniform lane failure as a capability gap. Check the grader first — the pnpm.cmd exit-127 trap made an entire UI lane (3/4) falsely fail and read as "the agent can't do UI."
• Treating a deflated category score as a capability gap before ruling out oracle-strictness. A trivially correct answer (easy mean, easy kv-get) that fails on numeric precision or JSON key order reads identically to a genuine capability failure. A 52 pp gap between strict and normalized scoring was measured in the wild (Ganapaneni 2026). Normalize the comparison first (strip/case/round/sort-keys/set-compare) and audit easy tasks that fail uniformly BEFORE drawing a capability conclusion. "Tool-use 40%" may be "40% minus N pp format deflation."
• Blending judge and oracle into one headline number. Judge variance pollutes the gate; quarantine it.
• Single-trial judging on a gate. Pairwise preferences flip 13.6% on average; 11 trials are needed for 95% fidelity. Use N≥3, randomized AB/BA, Cohen's κ — not raw agreement (which overstates discrimination 33–41 pp).
• Write-only-at-end. A crash or a killed subprocess loses every completed task. Checkpoint per task.
• Optimizing before validating the signal. A loop built on an unvalidated oracle gets worse at the real task while the metric appears to improve.