ablate-review-plan

name	ablate-review-plan
description	Ablation test harness for the review-plan skill. Runs the structured question-based control (SKILL.md) and the directive-based ablated variant (SKILL-v-ablation-na.md, per-directive N/A semantics) against the same fixture set with k=3 repetition, then uses the review-plan-ablation-judge to compare outputs for logical equivalence and reports per-fixture stability. Answers: do the structured question IDs add meaningful signal, or does a directive prompt achieve equivalent issue detection? Spot-Check 1 Pass Criterion (input3b — clean-plan calibration anchor): - Majority winner = TIE (or SPLIT, treated as TIE) - Mode verdict_agreement = EQUIVALENT - Verdict stability = VERDICTS_STABLE (all 3 control runs same verdict AND all 3 ablated runs same verdict) - Winner stability = WINNER_STABLE OR WINNER_UNSTABLE (the latter is acceptable iff verdict-stability is VERDICTS_STABLE AND mode false_positives ∈ {EQUIVALENT, CONTROL}; surface in the false-positive summary as a signal, not a fail) - Mode false_positives in {EQUIVALENT, CONTROL} (ablated must not over-flag more than control; false_positives = X means side X had more FPs, so the allowed set excludes the over-flagging side) - >= 2 of 3 control AND >= 2 of 3 ablated runs reach PASS Tolerates one stochastic flip per side; detects systematic over-flagging via the false_positives mode. Brittle "all 3 PASS" rule replaced after probe-9/input3 calibration repair (see RESULTS.md 2026-05-02 entries). AUTOMATICALLY INVOKE when user mentions: - "ablation test", "ablate review-plan", "test directive variant" - "does the question structure matter", "directive vs questions"
argument-hint	[--fixtures <probe-N,...|input-N,...>] [--single <fixture-short-name|plan-path>] [--variant <name>] [--model opus|sonnet|haiku] [--skip-cross-model] [--pre-reg <file>]
allowed-tools	Agent, Bash, Read, Glob, Write, Edit

ablate-review-plan Skill

Measures whether the structured question-based control (plugins/review-suite/skills/review-plan/SKILL.md) produces materially better issue detection than the directive-based ablated variant (plugins/review-suite/skills/review-plan/variants/SKILL-v-ablation-na.md — per-directive N/A semantics, v2).

Architecture (v2 — k=3): For each fixture, run control × 3 and ablated × 3 in parallel, then run the judge × 3 (paired by index) once outputs land. Aggregate into a per-fixture majority winner + stability flag + per-criterion mode. The judge evaluates for logical equivalence (same issues found, same verdict) — not output quality. k=3 distinguishes stochastic noise from real regressions.

Spot-Check 1 failure-mode branch table (pre-registered):

The original v2 plan anticipated only false_positives = ABLATED. input3b hit false_positives = CONTROL + WINNER_UNSTABLE on identical verdicts — a mode the plan didn't enumerate, which forced a mid-flight retroactive criterion reshape. To prevent recurrence, the full branch space is enumerated before spot-checks 2/3 run:

Outcome	Interpretation	Action
All criteria EQUIVALENT, both PASS, WINNER_STABLE TIE	Clean PASS	Proceed
false_positives = ABLATED (mode)	Ablated over-flags clean plans	Patch ablated variant; do not proceed
false_positives = CONTROL (mode) + both PASS	Control over-flags clean plans (filed as separate finding)	Note in RESULTS.md + commit body; proceed (control defect, not bench failure)
Either side fails PASS-quorum (<2/3)	Verdict instability	Inspect raw outputs; if fixture-defect, reclassify; if model-stochasticity, retry with k=5
false_negatives = CONTROL on Gate-1 fixture	Coverage gap in directive set	Patch ablated; do not proceed
verdict_agreement = CONTROL or ABLATED (mode)	Substantive verdict divergence	Inspect; the direction of divergence determines branch

Pre-registration matters: deciding the branch after seeing the result is how plans get retroactively reshaped to fit the data. Step 5 of the prior calibration-repair plan did exactly that when it hit false_positives = CONTROL; this table closes that loophole.

---

Decision gate (pre-registered, v2 — requires stability):

Result	Interpretation
≥80% majority TIE/ABLATED + ≥80% VERDICTS_STABLE + 0 Gate 1 false negatives	Recommend replacement — ablated v2 is sufficient
≥2 Gate 1 false negatives (incl. hidden-issue probes 17–21)	Questions are load-bearing — keep structured evaluation
<80% VERDICTS_STABLE	Variant is stochastically unreliable — not deployable regardless of mean performance
Mixed (false negatives on advisory but not Gate 1)	Partial — Gate 1 directives load-bearing; trim advisory directives

---

Step 0: Parse Arguments

Read the invocation arguments. Supported forms:

/ablate-review-plan                                        # full 16-fixture suite × k=3
/ablate-review-plan --single probe-9                       # single fixture × k=3
/ablate-review-plan --single ~/.claude/plans/foo.md        # arbitrary plan-path × k=3 (canary mode)
/ablate-review-plan --fixtures probe-1,probe-9,input3      # comma-separated subset × k=3
/ablate-review-plan --variant null                         # use SKILL-v-null.md as ablated variant
/ablate-review-plan --variant micro-noclose-strict         # default — SKILL-v-micro-noclose-strict.md (v5.3, 2026-05-09)
/ablate-review-plan --model sonnet                         # default model for review agents
/ablate-review-plan --model opus --single probe-9          # cross-model regression spot check
/ablate-review-plan --skip-cross-model                     # skip Step 2.5 (CI cost saver)
/ablate-review-plan --pre-reg <file>                       # single-command harness; see Step 0c

--model <name> flag. Selects the model for both control and ablated review agents in Step 2a/2b. Valid: opus, sonnet, haiku. Default: sonnet. The judge model is independent and pinned to sonnet regardless — judge variance is not what we are measuring. Reject any other value with a hard error in this step. Persist as $RESULTS_DIR/run-config.json so post-hoc readers can tell which model produced which outputs.

--skip-cross-model flag. Boolean (default false). Skips Step 2.5 (the probe-9 cross-model regression cell) entirely. Auto-implied when --single resolves to a non-probe-9 fixture, when --fixtures does not include probe-9, or when --variant null (the senior-engineer baseline is not subject to the count-based-rule expectation).

--variant <name> flag. Selects which file in plugins/review-suite/skills/review-plan/variants/ is used as the ablated variant in Step 2b. The file resolved is plugins/review-suite/skills/review-plan/variants/SKILL-v-<name>.md. Default: micro-noclose-strict. Available variants:

--variant value	File	Description
micro-noclose-strict (default)	SKILL-v-micro-noclose-strict.md	v5.3 (default, 2026-05-09) — v5-micro-noclose + always-on count-based severity rule (zero/one/two+ blocking → PASS/NEEDS_UPDATE/NOT_READY). 5-fixture k=5 Sonnet ablation: strictly better-or-equal to micro-2close on every cell; wins on probe-9 and input11 via finding-count-aware downgrade. (≤35 hard gate)
micro-2close	SKILL-v-micro-2close.md	v5.2 Arm C (legacy default through 2026-05-09) — v5-micro-prose with 3 close categories folded into prose; keeps fabricated-quant + phantom-types close questions (≤34 hard gate)
micro-1close	SKILL-v-micro-1close.md	v5.2 Arm B — v5-micro-prose with 4 close categories folded into prose; keeps only fabricated-quant close question (≤32 hard gate)
micro-noclose	SKILL-v-micro-noclose.md	v5.2 Arm A — v5-micro-prose with all 5 close categories folded into directive prose; no Adversarial Close (≤28 hard gate)
micro-prose	SKILL-v-micro-prose.md	v5.1 Arm A — v5-micro + directive prose expanded to include advisory categories (≤40 hard gate)
null	SKILL-v-null.md	Senior-engineer one-liner baseline (Phase B)

The control side always uses plugins/review-suite/skills/review-plan/SKILL.md. Only the ablated side varies.

Fixture short-name → path map:

Short name	Path	Expected finding (ground truth)
probe-1	plugins/review-bench/fixtures/probes/probe-1-unvalidated-constraint.md	Q-G1: asserts PropertiesService rejected without benchmarks
probe-2	plugins/review-bench/fixtures/probes/probe-2-phantom-code-references.md	Q-G11: no file paths or function names cited
probe-3	plugins/review-bench/fixtures/probes/probe-3-cross-phase-contradiction.md	Q-G21/G22: cross-phase contradiction + undefined field
probe-7	plugins/review-bench/fixtures/probes/probe-7-untestable-verification.md	Q-G20: verification section has no runnable commands
probe-9	plugins/review-bench/fixtures/probes/probe-9-g1-pass-calibration.md	ambiguous-plan calibration — both versions should flag substantive issues; judge for symmetry, not PASS (see RESULTS.md 2026-05-02 spot-check 1)
probe-16	plugins/review-bench/fixtures/probes/probe-16-gas-chatservice-wrapper.md	Q-G21: internal contradiction in step scoping
probe-17	plugins/review-bench/fixtures/probes/probe-17-untrusted-log-injection.md	Hidden issue: untrusted X-Request-Id header → log injection (read first-line <!-- expected-finding: ... -->)
probe-18	plugins/review-bench/fixtures/probes/probe-18-silent-type-mismatch.md	Hidden issue: cited fn returns User | undefined, plan destructures unconditionally
probe-19	plugins/review-bench/fixtures/probes/probe-19-live-entry-point-removal.md	Hidden issue: removed function is a registered scheduled trigger (live external entry point)
probe-20	plugins/review-bench/fixtures/probes/probe-20-silent-async-rejection.md	Hidden issue: fire-and-forget void notifySignup(...) silently swallows rejection
probe-21	plugins/review-bench/fixtures/probes/probe-21-procedurally-clean-false-claim.md	Hidden issue: 10× speedup citation references a benchmark file no plan step produces
input3	plugins/review-bench/fixtures/inputs/input3-trivial-plan.md	mixed-defect calibration — both versions should flag malformed ## Git Lifecycle1. header + placeholder script ID + vacuous verification symmetrically (EQUIVALENT mode); not a clean-plan baseline (see RESULTS.md 2026-05-02 spot-check 1 retry)
input3b	plugins/review-bench/fixtures/inputs/input3b-trivial-pass.md	PASS calibration — neither version should flag anything substantive (true clean-plan baseline; one-line CLAUDE.md doc edit cited verbatim with runnable verification)
input4	plugins/review-bench/fixtures/inputs/input4-plan-with-issues.md	Structural problems (diverse)
input6	plugins/review-bench/fixtures/inputs/input6-node-refactor-missing-prereads.md	Phantom code references (Node.js)
input8	plugins/review-bench/fixtures/inputs/input8-gas-oauth-tbd-markers.md	Unresolved TBD markers
input11	plugins/review-bench/fixtures/inputs/input11-node-parallel-phases.md	Complex parallel phases

For probes that ship with a top-of-file <!-- expected-finding: ... --> HTML comment (probes 17–21), read the comment text and use it verbatim as the EXPECTED_FINDING for the judge. The comment is the ground-truth target finding.

Default fixture set (no args): all 17 above.

Path-mode resolution for --single (canary support). If the --single argument contains / or ends in .md, treat it as a literal path to a real plan rather than a short name. Resolve ~ to $HOME. The resolved file becomes the fixture for k=3. Generate a synthetic short name by slugifying the basename (strip .md, replace non-alphanumerics with -) for use in <fixture>-control-<i>.md / <fixture>-ablated-<i>.md output paths. EXPECTED_FINDING is empty (no ground-truth target). The Step 0b inspector runs as normal, but its suitable_as is informational only — there is no fixture-map expectation to compare against in path mode. Reject the path if it lives under plugins/review-bench/fixtures/{probes,inputs}/ (those are calibration fixtures, not real plans).

Recommended verification order:

1. Start with --single input3b (true PASS calibration — confirms harness wiring + over-flagging dimension before full run)
2. Then --single probe-17 (hidden-issue detection sanity) and --single probe-1 (k=3 stability sanity)
3. Then full suite

---

Step 0c: Pre-Registration Mode (--pre-reg <file>)

When --pre-reg <path> is supplied, this section runs before Step 0b and overrides --fixtures, --single, --variant, and --model with values from the pre-registration file. Hard-fail if any of those flags are also supplied — pre-registrations are immutable contracts; CLI override would defeat the point. --skip-cross-model is honored as an opt-out.

Pre-registration file schema (YAML frontmatter on top of narrative):

---
schema_version: 1
run_id: <unique-id>
arms:
  control:   { skill: <path-to-skill.md> }
  treatment: { skill: <path-to-variant.md> }
fixtures:
  - { id: <short-name>, path: <absolute-or-repo-relative-path> }
k: <int>
model: <opus|sonnet|haiku>
scoring:
  method: concept_grep_plus_verdict_tier   # only supported in v1
criteria:
  - id: <int>
    fixture: <short-name>
    concept_regex: <optional pcre>
    rule: "<predicate referencing rates>"   # human-readable; mechanical evaluation in Step 3-prereg
---

The narrative below --- is preserved verbatim and surfaced in the run summary.

Parsing. Use awk '/^---$/{c++; next} c==1' <file> to extract the frontmatter block. Prefer yq if available; otherwise parse the minimal subset above with awk/grep (the schema is intentionally flat so a shell parser suffices). Populate the same internal vars (FIXTURES, MODEL, VARIANT_PATH, K, CONTROL_SKILL_PATH) the existing pipeline uses; subsequent steps then run unchanged.

Parallelism cap. Cap dispatched agents at 10 concurrent. For a k × |fixtures| × 2 plan that exceeds 10, dispatch in batches of 10 via parallel Agent tool uses, await all in batch, then dispatch the next batch.

Per-agent error handling. Retry once on timeout / empty output / UNKNOWN verdict from score-run.sh. After one retry, mark the cell ERROR and continue. Halt only on: pre-reg parse error, fixture path not found, variant file not found, or >20% of dispatched agents in ERROR.

Step 3-prereg: scoring & report. Replace Steps 3/4 of the standard pipeline with:

1. For each run output $RESULTS_DIR/<fixture>-<arm>-<i>.md, invoke score-run.sh --output <file> --concept-regex <regex-from-criterion> (or --verdict-only if the criterion has no concept_regex). Accumulate per-cell rates: <numerator>/<k> for verdict-tier hits and concept hits.
2. Mechanically evaluate each rule predicate against the accumulated rates. Express treatment.NOT_READY_rate >= 4/5 as a literal numerator/denominator comparison.
3. Emit a markdown table mirroring RUN-FINDINGS.md "Per-cell against pre-registered criteria" format: one row per fixture, columns = criteria, cell content = <n>/<k> ✓ or <n>/<k> ✗ or E/<k> for error cells.
4. Write the table + the preserved narrative to $RESULTS_DIR/RUN-RESULTS.txt (filename matches RUN-FINDINGS.md convention).
5. Exit 0 if all criteria pass, exit 1 otherwise (so CI can gate on it).

Smoke test. A minimal pre-registration ships at plugins/review-bench/skills/ablate-review-plan/smoke-prereg.md reproducing the probe-9 row of RUN-FINDINGS.md. Acceptance: treatment.NOT_READY_rate >= 4/5 AND control.NEEDS_UPDATE_rate >= 4/5 (±1 cell stochastic tolerance for Sonnet variance). Cost ≈ 10 agents × ~1 min.

---

Step 0b: Fixture validity precondition

Before spawning any review agent, for each fixture in the active set, spawn ONE inspector agent (general-purpose, ~30s) with prompt:

Read this plan: <fixture path>. List every objectively defective element you can
identify (placeholder identifiers, malformed Markdown, vacuous verification steps,
contradictions, fabricated citations, missing pre-reads). Return JSON:
{defects: [...], suitable_as: 'PASS_calibration' | 'mixed_defect' | 'hidden_issue' | 'general'}

Compare the inspector's suitable_as to the fixture map's expected role (see Step 0 fixture map). If they disagree (e.g., inspector says mixed_defect, map says PASS_calibration), HALT and report — do not waste 6 agents + 3 judges on a known-broken fixture.

Why this exists: the recurring failure mode across probe-9 and input3 (each cost a full discovery cycle of 6 agents + 3 judges before the mis-classification surfaced) was that the harness ran the full pipeline and then discovered the fixture was mis-classified. Cost ratio: ~30s × N (one inspector per fixture) vs ~5min × N + 9 agents per discovery cycle on a bad fixture. The pre-flight is dominantly cheaper unless every fixture in the active set is correctly classified, in which case it costs ~30s × N up front.

---

Step 1: Setup

RESULTS_DIR=$(mktemp -d /tmp/ablate-review-plan.XXXXXX)
MODEL="${model_arg:-sonnet}"        # one of: opus, sonnet, haiku
echo "Results dir: $RESULTS_DIR"
echo "Model under test: $MODEL"
cat > "$RESULTS_DIR/run-config.json" <<EOF
{"model": "$MODEL", "variant": "$VARIANT", "fixtures": "$FIXTURES", "k": 3}
EOF

k=3 repetition: for each fixture, run control × 3 + ablated × 3 (six agents per fixture, all parallel-eligible). The judge compares pair-by-index — (control-1, ablated-1), (control-2, ablated-2), (control-3, ablated-3) — producing 3 judge JSONs per fixture. Stability across the 3 runs is reported alongside the majority winner in Step 3.

For each fixture in the active set, create temp file paths for k=3 runs:

• $RESULTS_DIR/<fixture>-control-{1,2,3}.md — control review outputs (3 runs)
• $RESULTS_DIR/<fixture>-ablated-{1,2,3}.md — ablated review outputs (3 runs)
• $RESULTS_DIR/<fixture>-judge-{1,2,3}.json — judge verdicts (one per paired run)

Variant under test: the ablated variant defaults to plugins/review-suite/skills/review-plan/variants/SKILL-v-micro-noclose-strict.md (v5.3 — promoted 2026-05-09 after 5-fixture k=5 Sonnet ablation; replaced micro-2close as production default). Use --variant <name> to point at any other file in plugins/review-suite/skills/review-plan/variants/ (resolved as SKILL-v-<name>.md). The file selected here is what gets passed as the system prompt to all 3 ablated-side agents in Step 2b. The control side always uses plugins/review-suite/skills/review-plan/SKILL.md.

---

Step 2: Run Each Fixture (k=3)

For each fixture, run Steps 2a and 2b for each repetition i ∈ {1,2,3}. The 6 runs per fixture (3 control, 3 ablated) are mutually independent — spawn them as parallel Agents. Once outputs are written, run Step 2c (judge) per index pair (also parallelizable across indices once both that pair's outputs exist).

Step 2a: Control runs (i=1,2,3)

For each i ∈ {1,2,3}, spawn an Agent on model: ${MODEL} with the full text of plugins/review-suite/skills/review-plan/SKILL.md as the system prompt and the fixture file content as the input. Write the agent's complete review output to $RESULTS_DIR/<fixture>-control-<i>.md. All 3 control agents run on ${MODEL}.

Agent prompt template:

You are running the review-plan skill in control mode.

<SKILL>
[full contents of plugins/review-suite/skills/review-plan/SKILL.md]
</SKILL>

Review this plan:

<PLAN>
[full contents of fixture file]
</PLAN>

Output your complete review. Do not truncate.

Step 2b: Ablated runs (i=1,2,3)

For each i ∈ {1,2,3}, spawn an Agent on model: ${MODEL} with the resolved variant file (plugins/review-suite/skills/review-plan/variants/SKILL-v-<variant>.md, where <variant> is the value of the --variant flag, defaulting to micro-2close) as the system prompt and the same fixture content. Write output to $RESULTS_DIR/<fixture>-ablated-<i>.md.

Agent prompt template:

You are running the review-plan skill in ablated directive mode (per-directive N/A variant).

<SKILL>
[full contents of plugins/review-suite/skills/review-plan/variants/SKILL-v-ablation-na.md]
</SKILL>

Review this plan:

<PLAN>
[full contents of fixture file]
</PLAN>

Output your complete review. Do not truncate.

Step 2c: Judge (k=3, paired by index)

For each i ∈ {1,2,3}, read <fixture>-control-<i>.md and <fixture>-ablated-<i>.md. Spawn review-plan-ablation-judge agent (model pinned to sonnet regardless of --model — judge model is held constant so cross-model runs remain comparable) with:

<CONTROL_REVIEW>
[contents of $RESULTS_DIR/<fixture>-control-<i>.md]
</CONTROL_REVIEW>

<ABLATED_REVIEW>
[contents of $RESULTS_DIR/<fixture>-ablated-<i>.md]
</ABLATED_REVIEW>

<EXPECTED_FINDING>
[expected finding string from fixture map. For probes 17–21, read the top-of-file
`<!-- expected-finding: ... -->` HTML comment from the probe file and use the comment
text verbatim. Otherwise pass empty string.]
</EXPECTED_FINDING>

Parse the single-line JSON response. Write it to $RESULTS_DIR/<fixture>-judge-<i>.json.

The 3 judge calls per fixture are mutually independent — run them in parallel as soon as both outputs for the corresponding index exist.

Parallelism: all 6 control/ablated agent runs for a given fixture, and all 3 judges, are independent and parallel-eligible. Across fixtures, the runs are also independent. The upper bound on parallelism is your Agent capacity, not data dependencies. Within capacity, prefer to dispatch all runs in a single message of parallel Agent tool uses, then dispatch all judges in a second parallel batch once outputs land.

---

Step 2.5: Cross-Model Regression Cell (probe-9)

Runs unconditionally on every full-suite invocation. Skipped when:

• --skip-cross-model is supplied,
• --single resolves to a fixture other than probe-9,
• --fixtures does not include probe-9,
• --variant null is selected (the senior-engineer baseline is not subject to the count-based-rule expectation).

Cell shape (4 runs, k=1, no judges — a tripwire, not a comparison):

Cell	Model	Side	Reps
A	opus	control (SKILL.md)	1
B	opus	ablated (resolved --variant)	1
C	sonnet	control	1
D	sonnet	ablated	1

Dispatch four agents in parallel with the probe-9 fixture as input. Write outputs to $RESULTS_DIR/cross-model-{opus,sonnet}-{control,ablated}.md.

Verdict extraction. For each cell, grep the agent output for PASS, NEEDS_UPDATE, or NOT[ _]READY in the verdict line.

Expected baseline (RUN-FINDINGS.md, 2026-05-09):

expected = {
  (opus,   control): NOT_READY,     # prior-research baseline
  (opus,   ablated): NOT_READY,     # treatment was strictly ≥ control on probe-9
  (sonnet, control): NOT_READY,     # verified live 2026-05-09: control unified with ablated post-micro-noclose-strict; prior NEEDS_UPDATE deprecated
  (sonnet, ablated): NOT_READY,     # treatment's count-based rule scales
}

Verdict computation: the cell PASSes iff observed[(model, side)] == expected[(model, side)] for all four cells. Otherwise FAIL loudly, naming each off-cell with observed/expected.

Allowed cross-model gap. As of 2026-05-09, all four cells are expected NOT_READY — the prior Opus → NOT_READY / Sonnet → NEEDS_UPDATE control divergence was deprecated when the control SKILL.md was unified with the ablated arm post-micro-noclose-strict promotion. Any tier divergence from the expected baseline table above fails the regression cell. Specifically:

• Either model dropping to PASS on probe-9 control is a hard fail (the directive prose stopped firing).
• Sonnet ablated dropping below NOT_READY is a hard fail (the count-based severity rule regressed).
• Opus control dropping below NOT_READY is a hard fail (prior-research baseline regressed).
• Haiku is informational only and not part of this baseline; running with --model haiku is allowed but emits INFORMATIONAL only.

Update this expected table only with a paired RUN-FINDINGS.md entry justifying the new baseline; do not silently relax.

---

Step 3: Aggregate Results (k=3 majority + stability)

After all fixtures complete, read all judge JSON files and compute per-fixture aggregates across the 3 paired runs.

Per-fixture aggregation

For each fixture, collapse the 3 judge JSONs into:

• Majority winner: the most common value of winner across the 3 judges. Examples: [CONTROL, TIE, CONTROL] → CONTROL; [TIE, TIE, ABLATED] → TIE; [CONTROL, TIE, ABLATED] → no majority — record as SPLIT and treat as TIE for decision-gate aggregation.
• Verdict stability: VERDICTS_STABLE if (a) all 3 control runs reach the same final verdict (PASS / NEEDS_UPDATE / NOT READY) AND (b) all 3 ablated runs reach the same final verdict. This is the substantive signal — it answers "does the variant produce reliable verdicts." VERDICTS_UNSTABLE otherwise.
• Winner stability: WINNER_STABLE if all 3 judges agree on winner; WINNER_UNSTABLE otherwise. This is a secondary tiebreak; can be UNSTABLE even when both sides issue identical verdicts (judges disagree on whose extra-flagging counts as the better calibration — see input3b 2026-05-02 for the canonical case).
• Per-criterion mode: for each of the 5 criteria (issue_overlap, false_negatives, false_positives, severity_alignment, verdict_agreement), the mode value across the 3 judges. If all three differ on a criterion, record SPLIT.

Per-fixture winner + stability table

Fixture	Majority Winner	Verdict Stability	Winner Stability	issue_overlap	false_negatives	false_positives	severity_alignment	verdict_agreement
probe-1	?	VERDICTS_STABLE/UNSTABLE	WINNER_STABLE/UNSTABLE	?	?	?	?	?
...

Per-criterion win rates

For each criterion, count across all fixtures using the per-criterion mode:

• Control wins: N
• Ablated wins: N
• Equivalent: N

Stability summary

• Total fixtures: N
• VERDICTS_STABLE: N (%) ← substantive signal: verdict reproducibility
• VERDICTS_UNSTABLE: N (%)
• WINNER_STABLE: N (%) ← secondary tiebreak signal
• WINNER_UNSTABLE: N (%)

A high VERDICTS_UNSTABLE rate (>20%) is a signal that the ablated prompt is not robust enough to deploy regardless of mean performance. WINNER_UNSTABLE alone is acceptable iff verdict-stability is VERDICTS_STABLE and mode false_positives ∈ {EQUIVALENT, CONTROL}; surface it in the false-positive summary as a signal, not a fail.

False negative summary

List each fixture where the mode of false_negatives is CONTROL, with the expected finding and a one-line note on what the ablated review missed. For probes 17–21, the expected finding comes from the probe's top-of-file <!-- expected-finding: ... --> comment.

Overall verdict (decision gate v2 — requires stability)

Apply the pre-registered decision gate. The recommendation that the ablated variant should replace the structured skill in production requires all three conditions:

total_fixtures        = len(active_fixtures)
tie_or_ablated_count  = count(majority_winner ∈ {TIE, ABLATED})
stable_count          = count(verdict_stability == VERDICTS_STABLE)   # measures verdict reproducibility — what the 80% gate was always trying to measure
fn_count              = count(mode(false_negatives) == "CONTROL")

# Gate 1 probes — original L1 probes plus the 5 hidden-issue probes from v2
gate1_probes = {probe-1, probe-2, probe-3, probe-7, probe-9, probe-16,
                probe-17, probe-18, probe-19, probe-20, probe-21}
gate1_fn_count = count(mode(false_negatives) == "CONTROL" AND fixture IN gate1_probes)

if (tie_or_ablated_count / total_fixtures >= 0.80
    AND stable_count / total_fixtures >= 0.80
    AND gate1_fn_count == 0):
    verdict = "RECOMMEND_REPLACEMENT — ablated v2 matches/beats control with stable verdicts and zero Gate 1 false negatives"
elif gate1_fn_count >= 2:
    verdict = "CONTROL_BETTER — Gate 1 hidden-issue probes regress under ablated v2 (≥2 false negatives)"
elif stable_count / total_fixtures < 0.80:
    verdict = "UNSTABLE — variant prompt is stochastically unreliable; not deployable regardless of mean performance"
elif fn_count >= 2 AND gate1_fn_count == 0:
    verdict = "MIXED — Gate 1 questions load-bearing; advisory directives candidates for trimming"
elif fn_count == 1:
    verdict = "MIXED — single false negative; review manually before deciding"
else:
    verdict = "INCONCLUSIVE — no Gate 1 false negatives but TIE/ABLATED rate below 80% threshold"

---

Step 4: Output

Print the aggregate table, per-criterion win rates, false negative summary, and overall verdict.

After printing the aggregate table, also print a Judge Reasoning Digest: for each fixture, the 3 judge reasoning strings concatenated with their winner labels and key criteria modes. Save to: $RESULTS_DIR/judge-reasoning-digest.md.

Format per fixture:

## <fixture> — majority winner: <X>; verdict_stability: <Y>; winner_stability: <Z>

### Judge 1 (winner=<W1>; false_positives=<FP1>; false_negatives=<FN1>)
<reasoning string>

### Judge 2 (winner=<W2>; false_positives=<FP2>; false_negatives=<FN2>)
<reasoning string>

### Judge 3 (winner=<W3>; false_positives=<FP3>; false_negatives=<FN3>)
<reasoning string>

This is the human-readable signal that complements the criteria modes. Across probe-9, input3, and input3b, the judges' reasoning text — not their JSON criteria alone — has been the most informative discriminator. Concrete example: input3b judges 1+3 surfaced the Q-E2 control-over-flagging finding (criterion false_positives = CONTROL alone flagged the direction of over-flagging but not what the control was over-flagging on — "Post-Implementation Workflow self-edit on a one-line doc plan" lives only in the reasoning text). Promoting it to a first-class output (not buried in 3 JSONs per fixture) shortens the spot-check feedback loop.

Then print:

Results saved to: $RESULTS_DIR
Reasoning digest:  $RESULTS_DIR/judge-reasoning-digest.md

Cleanup is NOT automatic — leave the results dir so the user can inspect raw outputs.

---

Manual Verification Checklist

After running the full suite, verify:

1. input3b (PASS calibration — true clean-plan anchor): the Spot-Check 1 Pass Criterion (top-of-file) applies — majority winner TIE, mode verdict_agreement EQUIVALENT, VERDICTS_STABLE (WINNER_UNSTABLE tolerated under the conditions in the criterion), mode false_positives ∈ {EQUIVALENT, CONTROL}, and ≥ 2 of 3 control AND ≥ 2 of 3 ablated runs reach PASS. Mode false_positives == ABLATED here means the ablated variant is over-flagging clean plans (likely tighten Q-G6/Q-G7 N/A clauses for one-line doc edits); mode false_positives == CONTROL means the structured control over-flags trivial plans (filed as a control defect — see RESULTS.md 2026-05-02 input3b finding for the Q-E2 case).

   input3 and probe-9 (mixed-defect / ambiguous-plan calibration — symmetry-of-flagging tests, NOT PASS baselines): both control and ablated should land at NEEDS_UPDATE-or-worse with overlapping issue clusters; the per-fixture mode of verdict_agreement should be EQUIVALENT and majority winner TIE. These fixtures test that both variants flag substantive defects (input3: malformed Git Lifecycle header + placeholder script ID + vacuous verification; probe-9: fabricated benchmark citation + dual-write ambiguity + undefined Memory type) symmetrically, not that either side issues PASS (see RESULTS.md 2026-05-02 entries for diagnoses).

2. probe-1 (unvalidated constraint): the ablated review should flag the unsubstantiated PropertiesService claim in at least 2 of 3 runs. Mode false_negatives == "CONTROL" here means the per-directive N/A variant is missing the evidence-checking directive.

3. Hidden-issue probes (17–21): each must surface its targeted finding in at least 2 of 3 ablated runs. Inspect any probe where the mode is false_negatives == "CONTROL" — that's a coverage gap in the directive set the v2 variant is meant to close.

4. Read at least 2 fixtures where the mode of false_negatives is CONTROL side by side against the expected finding to confirm the judge is discriminating correctly.

5. Read at least 1 VERDICTS_UNSTABLE or WINNER_UNSTABLE fixture's three judge JSONs — confirm the disagreement reflects model stochasticity (not a deterministic harness bug like wrong fixture text being passed). For WINNER_UNSTABLE-only cases (verdicts identical on each side, judges differ on classifying stylistic differences), this is informational not blocking — see Spot-Check 1 Pass Criterion.