phase2-exit-check

name

phase2-exit-check

description

Exit rubric for Phase 2 (Small-Scale Verification). Load after the Analyst has written `phase_2_verification/analysis/analysis_{n}.md` against the latest finalized `result_{n}.md`, and the Main Agent needs to determine whether Phase 2 is done (proceed to Phase 3) or whether the run calls for a rollback. Evaluates structural completeness of the analysis only — the routing decision itself belongs to the Main Agent, which reads the dominant rollback depth this check surfaces. Does NOT re-diagnose results, does NOT auto-invoke Loop 3.

Phase 2 Exit Check

Read the latest phase_2_verification/analysis/analysis_{n}.md, its matched phase_2_verification/results/result_{n}.md, the confirmed phase_1_design/method_design_v{n}.md, and phase_0_ideation/ideas/IDEA.md. Apply the five criteria below in order. Each criterion is binary: pass or fail.

A single failing criterion means the overall result is NEEDS-REWORK.

This check is structural. It asks: did the Analyst produce a handoff-ready artifact? It does not re-diagnose the run, does not dispute severity assignments, and does not make the Loop 3 routing decision. The routing signal — the Analyst's "Dominant rollback depth" — is surfaced in the output for the Main Agent to act on, but is not itself a criterion.

---

Criterion 1 — Latest result is finalized and matched

Pass if the most recent result_{n}.md (highest {n}) has Status: COMPLETE or FAILED (not RUNNING), AND the analysis file being evaluated is analysis_{n}.md with matching {n}.

Fail if the latest result is still RUNNING — Phase 2 exit-check should not be invoked on a running experiment; the pipeline was misdriven. Halt and escalate.

Fail if the {n} indexes differ (e.g., analysis_3.md evaluated against result_2.md). The Analyst must work on the latest run.

---

Criterion 2 — Required sections present

Pass if analysis_{n}.md contains all of the following sections, in this order:

1. Header (with result path, spec path, IDEA path, run status, date)
2. Synthesis (with Overall verdict, Most informative next diagnostic, Residual uncertainty, Dominant rollback depth — all four lines present, none empty)
3. Claim-Result Comparison (table; at least one row)
4. Findings (may be omitted in full if there are zero findings AND the run was clean — in that case §Synthesis must explicitly state "No findings; proceed to Phase 3")
5. Context for the diagnosis (may be "No contextual caveats" but the section must be present)
6. Evidence not available (may be "All findings fully evidenced" but the section must be present)

Fail if any section is missing or is empty in a way the template did not anticipate. An explicit placeholder counts as content.

---

Criterion 3 — Every Core Claim covered

Pass if the Claim-Result Comparison table contains exactly one row per Core Claim in IDEA.md, AND every row's Verdict cell reads SUPPORTED, NOT SUPPORTED, or UNMEASURABLE (no blanks, no "TBD", no "partial").

Fail if any Core Claim from IDEA.md is missing from the table, or any row's Verdict is blank or uses a label outside the three allowed values.

To check: read IDEA.md Core Claims, enumerate each, verify each appears as a row with a valid Verdict.

---

Criterion 4 — Every finding carries evidence

Pass if every finding in §Findings (across all categories) has all four of: Observation, Evidence (with a specific citation — metric value from the result file, log line, spec passage, or Core Claim), Why-this-category, and Scope. Or §Findings is legitimately empty (Criterion 2's carve-out).

Fail if any finding is missing one of those four fields, OR any Evidence field is a vague gesture ("the results look bad", "the method seems off") without a specific citation.

---

Criterion 5 — Severity is consistent with claim verdicts

Pass if: if any Core Claim in the Claim-Result Comparison has Verdict NOT SUPPORTED, at least one finding in §Findings carries severity High or Critical. (A NOT SUPPORTED claim cannot be explained by Low or Medium severity alone — the Analyst is instructed to re-walk classification if this mismatch appears.)

Pass also if every Core Claim is SUPPORTED or UNMEASURABLE — in which case there is no severity-consistency requirement to check.

Fail if a NOT SUPPORTED claim is present but no finding is High or Critical. This is an Analyst-side inconsistency; the exit check flags it, but the fix is to re-invoke the Analyst, not to override the findings here.

---

Output

After evaluating all five criteria, output exactly this block:

## Phase 2 Exit Check

**Result: PASS / NEEDS-REWORK**

| Criterion | Result | Note |
|-----------|--------|------|
| 1. Latest result finalized and matched | PASS / FAIL | [what was found — index and status] |
| 2. Required sections present | PASS / FAIL | [what was found — list missing sections if any] |
| 3. Every Core Claim covered | PASS / FAIL | [what was found — list uncovered or invalid-verdict claims if any] |
| 4. Every finding carries evidence | PASS / FAIL | [what was found — list weak-evidence finding numbers if any] |
| 5. Severity consistent with claim verdicts | PASS / FAIL | [what was found — flag the inconsistency if any] |

**Failing criteria:** [list by number, or "none — all criteria passed"]

**Dominant rollback depth (from analysis, surfaced here):** None / Low / Medium / High / Critical

**Routing signal for the Main Agent (advisory, NOT a decision):**

| Depth | Typical Main-Agent target |
|-------|---------------------------|
| None | Proceed to Phase 3 (Main Experiments). |
| Low | Loop 3a — re-enter `phase2-implementation` Case B for the module the Analyst's Code-bug finding identified. |
| Medium | Revisit Experimental Setup Module or enter `phase1-method-design` Case B depending on whether the finding is "Bad setup" (→ Setup Module) or "Feasibility failure" (→ Setup Module; possibly spec revision if resource substitution is needed). |
| High | Loop 3b — enter `phase1-method-design` Case B to redesign the component, OR reframe the claim via a Refiner round on IDEA.md, depending on whether the finding is "Method component fails" vs. "Claim mismatch". |
| Critical | Phase 0 re-ideation — enter `loop1-ideation-pipeline` Case C after user confirmation. This is a Main-Agent-owned escalation; do not auto-route. |

**Findings snapshot (for routing context):** [One line per category that has findings, with count and severity. E.g., "Code bug: 1 Low; Claim mismatch: 1 High." Or "No findings — run is clean."]

**Open known risks (from context):** [From §Context-for-the-diagnosis: any Accepted Additions active during the run, any engaged Risk Flags. Or "None carried forward."]

---

What this skill does NOT do

• Does not re-diagnose the run. The Analyst's categorization and severity are authoritative for routing purposes. Criterion 5 flags structural inconsistency between severity and claim verdict; it does not second-guess the Analyst's category assignment.
• Does not auto-invoke Loop 3a, Loop 3b, or Phase 0. The Routing signal table is advisory. The Main Agent decides — and for Critical, user confirmation is required.
• Does not suggest how to fix failing structural criteria. If Criteria 1–5 fail on structure, the remedy is to re-invoke the Analyst with the specific gap named; that is the Main Agent's call, not this skill's.
• Does not compare to prior runs. Cross-run trend analysis (did Loop 3a actually fix the previous bug?) is a Main Agent concern that lives in the conversation, not in this rubric.

---

Loop 3 routing — the Main Agent's reading

Once this skill returns NEEDS-REWORK with a non-None rollback depth, the Main Agent's job (outside this skill) is to:

1. Read the Analyst's findings — specifically the category and scope of each High/Critical finding; these name the module, component, or claim to be revisited.
2. Pick the entry case — phase2-implementation Case B names one module; phase1-method-design Case B names the specific findings to address; experiment-setup-pipeline is re-entered when resources need revision.
3. Surface to the user before acting — Loop 3a on a small module may proceed without prompt; Loop 3b, Experimental Setup revision, and Phase 0 re-ideation all require explicit user confirmation, because they invalidate upstream artifacts.
4. Record the routing decision in _logs/decision_log.md with the result / analysis index, the chosen loop, and the reasoning — the Loop 3 decision is a governance moment, not a routine step.

This guidance is inside the exit-check skill because the exit check is the natural place for the Main Agent to load the routing table; it is not a separate orchestration skill.

For the Phase 0 re-ideation path specifically: there is no "Loop 3c" — the entry point is loop1-ideation-pipeline Case C (re-entry with existing IDEA, which may be revised or replaced). The Critical severity exists so the Main Agent does not silently route a fundamentally-invalid idea through another Phase 1/2 cycle.