experiment-setup-exit-check

name

experiment-setup-exit-check

description

Exit rubric for the Experimental Setup Module. Load when the Main Agent needs to determine whether EXPERIMENT_SETUP.md is complete enough to hand off to Phase 1 (Method Design). Returns PASS or NEEDS-REWORK with specific failing criteria listed. Evaluates structural completeness and verification quality only — not whether the chosen baselines are the 'right' ones (that judgment belongs to the Architect).

Experiment Setup Exit Check

Read phase_0_ideation/experiment_setup/EXPERIMENT_SETUP.md. Apply the four criteria below in order. Each criterion is binary: pass or fail. Do not score, weight, or partially credit them.

A single failing criterion means the overall result is NEEDS-REWORK.

This check is structural. It asks: did the auditor produce a usable artifact? It does not ask whether the auditor made the best choices — that is the Architect's problem at Phase 1.

---

Criterion 1 — Required sections present

Pass if the file contains all of the following sections in this order:

1. Header (with idea path, audit date, and source-literature file list)
2. Datasets
3. Metrics
4. Baselines
5. LLM Configuration
6. Available Code
7. Removed / Unavailable Items
8. Supplementary Searches

Fail if any of these is missing or empty in a way the template did not anticipate. An explicit empty-state note (e.g., "No verified datasets — source literature did not yield an obtainable option") counts as content for the purpose of this criterion. A blank section with no content and no empty-state note fails.

---

Criterion 2 — No empty primary categories

Pass if Datasets, Baselines, LLM Configuration, and Available Code each have at least one verified entry OR an explicit empty-state note placed by the auditor.

This criterion separates a usable audit from a failed one. The Architect needs either real items or an explicit signal that the source literature did not support a category.

Fail if any of those four categories is empty AND has no empty-state note.

Note: Metrics is allowed to be sparse. Many tasks have only one or two standard metrics, and the Architect can refine metric choice at Phase 1.

Important: "Empty with empty-state note" passes Criterion 2 but causes the exit check overall to surface the empty category to the Main Agent in the output block as a "thin coverage" finding. The Main Agent then chooses whether to supplement (per experiment-setup-pipeline Step 4) — but the pipeline itself treats this as a NEEDS-REWORK signal. To make that wiring clean: if any primary category has an empty-state note, set the overall result to NEEDS-REWORK with Criterion 2 still marked PASS but with the thin category recorded in the output's "Thin coverage" line.

---

Criterion 3 — Every listed item is verified

Pass if every entry in the main sections (Datasets, Baselines, LLM Configuration, Available Code) has either a verification date OR a verified access URL recorded inline.

Fail if any listed entry lacks both. An entry without verification has no business being in the main sections — it should have been moved to Removed / Unavailable.

To check: spot-check 2–3 entries per section. If any spot-check fails, do a full walk and list every offending entry.

---

Criterion 4 — Removed items have specific reasons

Pass if every row in the Removed / Unavailable Items table has a specific, actionable reason (e.g., "404 on download page on 2026-04-18", "HF repo gated, access denied", "GitHub repo deleted") — not vague text like "unavailable" or "could not access".

Pass also if the table is empty AND the auditor wrote "No candidates were removed during verification."

Fail if any row's reason is vague or missing.

This criterion protects the audit's auditability: the user and downstream agents must be able to see why something was excluded, in case the situation changes (e.g., a gated dataset becomes ungated, a deleted repo gets restored).

---

Output

After evaluating all four criteria, output exactly this block:

## Experimental Setup Exit Check

**Result: PASS / NEEDS-REWORK**

| Criterion | Result | Note |
|-----------|--------|------|
| 1. Required sections present | PASS / FAIL | [what was found] |
| 2. No empty primary categories | PASS / FAIL | [what was found] |
| 3. Every listed item is verified | PASS / FAIL | [what was found] |
| 4. Removed items have specific reasons | PASS / FAIL | [what was found] |

**Failing criteria:** [list by number, or "none — all criteria passed"]

**Thin coverage:** [list any primary category that has an empty-state note,
which forces the overall result to NEEDS-REWORK even when Criterion 2 passes —
e.g., "Datasets: empty-state note present, source literature did not yield
verified options". Or "none — all primary categories have verified entries".]

**Removed items summary:** [one line: count of removed items + the most
consequential category, e.g., "3 items removed (2 datasets, 1 baseline)".
Or "none removed."]

Do not suggest how to fix failing criteria or thin coverage. The experiment-setup-pipeline skill handles routing — supplement, accept, revise, or abandon — based on user decision.