skill-gauntlet

# Skill Gauntlet — regression-guarded A/B evaluation Adopt a candidate only when it beats the champion **on value dimensions and does not regress on guard dimensions**. Totals and preference votes measure added value; conjunctive guards protect existing value. Candidates that win every blind forced choice can still be correctly rejected — that is the harness working, not failing. Three inviolable rules: 1. **Pre-register everything before running any cell** — the adoption gate, the cells, the instruments, the comparison set. No goalpost movement after results arrive. If a bar turns out to be mis-set, report both readings; the pre-registered text decides. 2. **Blind everything scoreable.** Judges, auditors, and executors never see version names, the sealed mapping, or planted-flaw keys. Only the coordinator holds the mapping. 3. **The champion is frozen.** Never modify the champion artifact during a round, not even "harmless" fixes. Hash it (md5) before and after. ## The workflow ### Phase 0 — Scope and design - Define the **champion** (current shipped artifact) and the **candidate** (exact diff from champion; verify by `diff` that nothing else changed). - Design **briefs**: 2–4 realistic tasks hard enough that the skill should matter, in fresh domains the skill has never seen. For skills with detection claims, plant 3–4 flaws per brief and write the **key matrix** (see [references/key-verification.md](references/key-verification.md)). - Design **fixtures** (see [references/fixtures.md](references/fixtures.md)): a framing-invariance pair (same facts, hyped vs skeptical register) and a negative fixture (a task too small for the skill — the highest-information single cell; every evolution round ratchets toward more machinery, so price restraint). - Freeze the **rubric**: 7–9 dimensions split into value dims (where the candidate claims gains) and guard dims (where regression is fatal). Anchor every odd rung. See [references/judge-panel.md](references/judge-panel.md). ### Phase 1 — Pre-register the gate Write the adoption rule into the results file BEFORE any execution. Conjunctive template (adapt dims; full template in [references/protocol.md](references/protocol.md)): > Candidate passes iff ALL of: (1) ≥ champion on every guard dim, on every instrument; (2) value-dim margins ≥ pre-set floors; (3) key detection ≥ relative-to-champion AND ≥ 90% absolute floor; (4) framing invariant; (5) negative fixture behaves; (6) instrument checks pass (controls gap, inter-judge agreement). Anything less → champion ships unchanged, candidate's deltas ship only as a research note. Add the **symmetric-loss clause**: on failure the candidate branch is discarded; its deltas return to an inventory with failure data attached. Never "fix and re-judge the same run" — that is goalpost movement. ### Phase 2 — Execute cells blind Spawn one executor subagent per cell. Each executor gets only: the skill path (champion or candidate), the brief verbatim, an output path. No testing context, no keys, no version names. Cells: candidate × briefs × N runs (N≥2 for variance), champion × same, plus fixtures for both, plus 1 **generic control** per brief (no skill at all) — controls calibrate the discrimination floor. ### Phase 3 — Blind and leak-scan Copy raw outputs to anonymized codes via `scripts/normalize.py` (mapping JSON → blind codes; runs a leak regex scan; inspect hits manually, never auto-fix). Codes and judge panel assignments go in a coordinator-only sealed mapping file. ### Phase 4 — Score with three instrument classes Run all three in parallel; they are blind to each other and to the mapping. - **Judge panels** (2–3 persona-diverse judges, crossed design: every brief scored by ≥2 judges, every judge sees both arms mixed): dimension-major scoring, forced choice per brief, suspected-generic call. Prompt patterns and persona design: [references/judge-panel.md](references/judge-panel.md). - **Cross-check auditor**: re-scores the guard dims mechanically (rule-text applied literally), runs the framing and restraint gates. When judges and auditor disagree on a guard dim, **the conservative reading governs**. See [references/auditor.md](references/auditor.md). - **Key verifier**: grades planted-flaw detection (✓/~/✗ with evidence pointers, hard-grader rule: torn → lower grade). See [references/key-verification.md](references/key-verification.md). Pool judges with `scripts/scorecard.py` (judges must emit a machine-readable scores block — required by the panel prompt). ### Phase 5 — Apply the gate exactly Compute margins per dimension, check every conjunctive condition, write the verdict. On PASS: repackage/reinstall the candidate, write the round report. On FAIL: champion ships unchanged; write the research note capturing validated wins, failure autopsies, and the next candidate's specified fixes. Either outcome is a deliverable. ## Hard-won operating rules (each cost a round to learn) - **Dimension-major scoring** (score dim 1 across all reports, then dim 2…) is what makes inter-judge agreement reach ~100% cell-level within-1pt. Do not let judges score report-by-report. - **The negative fixture is the highest-information cell.** Skills drift toward ceremony; both champion and candidate once ran full machinery on a trivial task. Keep one in every round. - **Absolute 100% detection bars are usually unattainable** on freshly planted hard keys — pre-register relative-to-champion plus a 90% floor instead, and say so before running. - **Instruments will disagree.** Judges grade holistically; the auditor applies rule text mechanically. Both readings are real. Pre-commit to the conservative one for adoption decisions. - **Prose bans do not stop template-driven leaks.** If the report template is organized as numbered stages, executors will cross-reference "Stage 4" no matter how strongly the skill forbids it. Enforce with a mechanical pre-delivery grep gate, or de-ordinal the template. - **A compliance failure is not a design failure** — log which one you saw. A candidate whose mechanism works but whose self-check missed a case needs a checklist fix, not a re-mine. - **Do not chase a specific key.** If every arm misses the same planted flaw the same way across builds, document it as a hard cell; tuning the skill to that cell teaches to the test and corrupts the harness. - **Margins are not decision-grade without uncertainty.** With the small N these rounds run, pre-register an INCONCLUSIVE verdict (collect more cells) alongside PASS/FAIL, and treat any margin inside the noise band as "not yet evidence." Details in [references/protocol.md](references/protocol.md). - **Guard dims include operational ones when the artifact ships to production** — cost, latency, output-schema compatibility. A candidate that triples cost or breaks downstream consumers must fail the gate even when quality dims pass. - **Delete subagents after their results arrive; never spawn judges who can see each other's scores.** - **Normalize format tells in the blinder.** When arms differ in template format (numbered vs unnumbered headers), every judge can identify the arm at a glance. Strip the tell from ALL blind copies (`normalize.py --strip-h2-ordinals`) while leaving prose leaks intact — those are auditor evidence. - **A fixture is burned the moment its answer enters any candidate's text.** Quoting the expected fixture answer inside a skill (even as an example) means restraint on that fixture can never be measured again. Rotate fixtures every round; grep candidates for fixture content before running. - **Auditor prompts quote the pre-registered rule text verbatim — never paraphrase.** A paraphrase that adds one category ("tag or class names") can charge rubric-required content as violations and split your instruments; score any extra category separately as exploratory, never in the binding number. - **The cheap spread noise-band is direction-blind.** A candidate above the champion in every single cell can still read as "inside the band". Wherever cell-level dominance is plausible, pre-register a paired sign test alongside the spread heuristic. - **Blind codes, judge IDs, and rubric dimension names share one namespace across ALL rounds.** Keep them disjoint (no D5 code when D5 is a dimension; no J3 code when J3 is a judge).