| name | skeptic-panel |
| description | Use when a screening positive appears in the continuous autotrain loop and before any confirmation spend — convene three independent skeptic lenses (statistical power, prior evidence, mechanism plausibility) to attempt refutation; majority-refute files the candidate as rejected with reasons |
Skeptic panel (refutation before confirmation spend)
Overview
A screening positive is a claim to be refuted, not a result to be
confirmed. RC3 (docs/design/harness-evolution-architecture-review-20260809.md)
showed the loop rediscovering and rejecting the identical delta
0.3267→0.3828 in nine separate loops because nothing tried to kill the claim
before spending confirmation cycles. This skill runs that kill attempt.
When: any screening positive (sdlc_delivery.json with positive: true
and cycle_intent: screening), BEFORE the champion queue spends a confirm or
promote cycle on it.
Inputs
python scripts/query_evidence.py output for the candidate's
config_fingerprint, lever keys, and endpoint metric (durable prior
attempts across sessions).- The cycle's preflight verdicts — the
preflight field of the campaign's
sdlc_delivery.json / outputs/autoresearch/sdlc_delivery_ledger.jsonl
entry (verdicts from slm_training.autoresearch.preflight, see
docs/design/preflight-gates.md).
Panel protocol
Convene three independent skeptics. Each writes its verdict
(refuted / not_refuted) with reasons BEFORE reading the other two — no
anchoring, no shared drafting.
Skeptic 1 — statistical power lens
- Could this measurement have rejected the null at all? (n paired documents,
power_feasibility_report in slm_training.autoresearch.evidence_ledger.)
- Is the observed delta within the noise band of prior same-arm deltas in the
evidence ledger? Single-seed fixture n is noisy by construction.
- Was the measurement complete (
measurement_complete: true, no decode
timeouts diluting the metric)?
Skeptic 2 — prior evidence lens
- Has this
config_fingerprint + endpoint been attempted before? With what
outcome and power? An identical config already confirm_failed /
ship_rejected with adequate power is a refutation by record.
- Is this delta suspiciously identical to a previously rejected one (the RC3
signature)? Same magnitude + same endpoint + same lever family ⇒ presumed
rediscovery until shown otherwise.
- Do
warn preflight verdicts indicate underpowered prior attempts this run
did not add power to?
Skeptic 3 — mechanism plausibility lens
- Is there a causal story from the lever moved to the endpoint metric that
respects the repo's architecture invariants (constrained decoding is the
product; capability is never bought with parameters)?
- Could the delta be an artifact: control mismatch, eval-partition drift
(
version_stamp eval components differ), steps/seed jitter, or a latency
win masquerading as quality?
- Does the win survive the non-regression metrics (parse_rate,
binder_reference_f1)?
Decision rule
- Majority refute (≥ 2 of 3) ⇒ file the candidate as rejected: record the
panel's reasons in the delivery record (append to
reasons, keep
positive reclassified honestly), do not enqueue/confirm the champion, and
ensure the outcome lands in the evidence store so the preflight
prior_attempts check blocks the rediscovery next time.
- Otherwise ⇒ the candidate proceeds to the normal confirmation path with
the panel's doubts attached as reasons.
Forbidden
- Weakening any gate, bar, alpha, or minimum_effect to let a candidate pass.
- Softening a skeptic's lens because the delta "looks exciting".
- Skipping the panel because preflight already passed — preflight gates spend,
the panel gates belief.
- Re-running the measurement until it confirms (that is p-hacking, not
skepticism).
A rejected candidate closes an approach, never a goal.
