phase3-exit-check

name

phase3-exit-check

description

Exit rubric for Phase 3 (Main Experiments). Load after every declared matrix cell has a finalized result_{n}.md + analysis_{n}.md, and the final story-coherence-advisor checkpoint has produced a story_check_{k}.md. Returns PASS or NEEDS-REWORK with a routing-advisory table (proceed to Phase 4 vs. rollback target). Evaluates structural completeness of the matrix's analyses and narrative coherence signal — NOT the quality of the results, NOT the routing decision itself (that belongs to the Main Agent), NOT ablation design (that is Phase 4). Does NOT re-diagnose any cell's analysis.

Phase 3 Exit Check

Read every phase_3_main_experiment/analysis/analysis_{n}.md in the directory, their matched phase_3_main_experiment/results/result_{n}.md files, the confirmed phase_3_main_experiment/specs/framework_v{m}.md, the full set of phase_3_main_experiment/story_checks/story_check_{k}.md, and phase_0_ideation/ideas/IDEA.md. Apply the six criteria below in order. Each criterion is binary: pass or fail.

A single failing criterion means the overall result is NEEDS-REWORK.

This check is structural. It asks: has Phase 3 produced a handoff-ready matrix + narrative? It does not re-diagnose cells, re-dispute the Analyst's verdicts, re-grade the matrix against some external bar, or make the Phase 4 vs. rollback routing decision. The routing signal — the aggregate rollback depth across cells + the latest story_check verdict — is surfaced in the output for the Main Agent to act on, but is not itself a criterion.

---

Criterion 1 — Matrix is complete and every cell is finalized

Pass if the set of cells declared in framework_v{m}.md §Modules §Runner §Dependencies (the full matrix) matches, one-to-one, the set of finalized runs under phase_3_main_experiment/results/. "Finalized" means Status: COMPLETE or Status: FAILED — no RUNNING entries remain.

Fail if any declared cell has no result_{n}.md, OR any result_{n}.md is still RUNNING, OR there is a result_{n}.md that does not map to a declared cell (stray runs — the Main Agent should resolve before exit).

To check: parse the Runner's Dependencies to enumerate declared cells; parse each result_{n}.md's §Slice to identify which cell it covered; cross-reference.

---

Criterion 2 — Every cell has a matched analysis

Pass if for every finalized result_{n}.md there exists a phase_3_main_experiment/analysis/analysis_{n}.md with matching {n}.

Fail if any finalized result has no analysis, OR any analysis's {n} does not match a result.

---

Criterion 3 — Every analysis is structurally complete

Pass if every analysis_{n}.md satisfies the same structural checks as phase2-exit-check Criteria 2, 3, 4, 5 applied per cell:

• All required sections present (Header, Synthesis with four lines, Claim-Result Comparison, Findings or explicit "No findings", Context for the diagnosis, Evidence not available).
• Every Core Claim from IDEA.md appears as a row in the Claim-Result Comparison, with a Verdict of SUPPORTED / NOT SUPPORTED / UNMEASURABLE.
• Every finding carries Observation, Evidence (with citation), Why-this-category, and Scope.
• Severity is consistent with claim verdicts — any NOT SUPPORTED claim has at least one finding at severity High or Critical.

Fail if any single analysis fails any of those four sub-checks. List which analysis and which sub-check failed.

---

Criterion 4 — At least one Phase 3 story check exists

Pass if phase_3_main_experiment/story_checks/story_check_{k}.md exists for at least one {k}, AND the latest story_check (story_check_{K}.md with the highest {K}) was written after the latest analysis (i.e., its Analyses read field references the latest analysis_{N}.md). This proves the Story Advisor saw the final state of the matrix.

Fail if no story_check exists, OR the latest story_check predates the latest analysis (a stale narrative that doesn't reflect the final run).

---

Criterion 5 — Latest story check verdict is handoff-compatible

Pass if the latest story_check_{K}.md has Verdict COHERENT, OR Verdict FRACTURED with every fracture's reconciliation scoped to Phase 4 (ablation-attribution fractures) and explicitly marked as such in the story_check.

Fail if Verdict is FRACTURED with at least one fracture whose reconciliation is NOT a Phase-4-scoped ablation question — e.g., a claim-reframing fracture that requires Refiner intervention, or a method-extension fracture that requires an Architect round. These fractures belong upstream of Phase 4, not in Phase 3's handoff.

The distinction: Phase 4 can sharpen "which component gets credit" but cannot fix "we claimed more than we can measure" or "the method doesn't work on dataset X." The former is allowed to carry through; the latter must be resolved before Phase 3 exits.

---

Criterion 6 — Aggregate severity is consistent with claim verdicts across the matrix

Pass if: across all analyses in the matrix, if any analysis_{n}.md marks a Core Claim as NOT SUPPORTED for the cell it covers, at least one analysis in the matrix carries a severity High or Critical finding. (This is the matrix-level version of phase2-exit-check Criterion 5.)

Pass also if every analysis's every Core Claim row is SUPPORTED or UNMEASURABLE across all cells.

Fail if any analysis has a NOT SUPPORTED claim but no analysis in the matrix has a High or Critical finding. The fix is to re-invoke the Analyst(s) on the cells where the inconsistency appears — not to override.

Note: A claim may legitimately be SUPPORTED on some cells and NOT SUPPORTED on others (e.g., method works on HotpotQA but not on Musique). That is a matrix-level signal — surfaced under Findings snapshot below — and triggers Criterion 5's scrutiny of the latest story_check, not Criterion 6.

---

Output

After evaluating all six criteria, output exactly this block:

## Phase 3 Exit Check

**Result: PASS / NEEDS-REWORK**

| Criterion | Result | Note |
|-----------|--------|------|
| 1. Matrix complete and finalized | PASS / FAIL | [counts: declared cells / finalized results; list any running or stray] |
| 2. Every cell has matched analysis | PASS / FAIL | [list any result without analysis, or vice versa] |
| 3. Every analysis is structurally complete | PASS / FAIL | [list analyses that failed any sub-check; specify which sub-check] |
| 4. At least one Phase 3 story check exists | PASS / FAIL | [latest story_check index and date; flag if stale] |
| 5. Latest story check verdict is handoff-compatible | PASS / FAIL | [latest Verdict; if FRACTURED, list fractures and whether each is Phase-4-scoped] |
| 6. Aggregate severity consistent with matrix claim verdicts | PASS / FAIL | [any NOT-SUPPORTED claims without a High/Critical finding anywhere in the matrix] |

**Failing criteria:** [list by number, or "none — all criteria passed"]

**Aggregate rollback depth (max across all cells' analyses):** None / Low / Medium / High / Critical

**Matrix claim-verdict snapshot (per Core Claim, across all cells):**

| Core Claim | SUPPORTED in | NOT SUPPORTED in | UNMEASURABLE in |
|------------|--------------|------------------|-----------------|
| [claim 1] | [list of cells / analyses] | [list] | [list] |
| [claim 2] | ... | ... | ... |

[One row per Core Claim from IDEA.md.]

**Latest story check:** phase_3_main_experiment/story_checks/story_check_{K}.md
**Latest story check verdict:** COHERENT | FRACTURED (scoped) | FRACTURED

**Routing signal for the Main Agent (advisory, NOT a decision):**

| Depth | Typical Main-Agent target |
|-------|---------------------------|
| None (and story COHERENT or FRACTURED-scoped) | Proceed to Phase 4 (Ablation). |
| Low | `phase2-implementation`-style Case B scoped to Phase 3 code directory for the module the failing cell's analysis identified. Resume Phase 3 as Case C for affected cells after fix. |
| Medium | Revisit Experimental Setup Module (`experiment-setup-pipeline`) for resource-level fixes, OR `phase1-method-design` Case B for spec tweaks affecting the matrix — depends on whether the finding is "Bad setup" / "Feasibility" (→ Setup) vs. spec-refinement (→ Phase 1). Resume Phase 3 as Case C after. |
| High | Loop 3b — `phase1-method-design` Case B to redesign the component, OR Refiner round on `IDEA.md` to reframe the claim — depends on whether the finding is "Method component fails" vs. "Claim mismatch". In Phase 3, High is significant: a failed Main Result implies the method does not generalize beyond Phase 2's slice; user confirmation before retreating. |
| Critical | Phase 0 re-ideation — enter `loop1-ideation-pipeline` Case C after user confirmation. Multiple Core Claims failed across the matrix and no single component or claim-reframing would recover. Do not auto-route. |

**Narrative routing overlay:** If Criterion 5 failed (latest story check is
FRACTURED with at-least-one non-Phase-4-scoped fracture), the Main Agent
must address that fracture before Phase 4, even if Aggregate rollback
depth is None. The Story Advisor's "Most important reconciliation" line
names the target (Refiner / Architect / supplementary matrix cell).

**Findings snapshot (for routing context):** [One line per analysis with
non-None rollback depth: "analysis_{n}.md: {category} {severity}". Or
"No non-None-depth analyses — matrix is clean."]

**Open known risks (from context across analyses):** [Union of Accepted
Additions and engaged Risk Flags carried forward across the matrix. Or
"None carried forward."]

---

What this skill does NOT do

• Does not re-diagnose any cell. Each analysis's categorization and severity are authoritative for routing purposes. Criterion 3 checks structural completeness per cell; Criterion 6 checks matrix-level severity consistency. Neither second-guesses the Analyst's category assignments.
• Does not auto-invoke Phase 4, Loop 3, or Phase 0. The Routing signal table is advisory. The Main Agent decides — and for High and Critical, user confirmation is required.
• Does not dispute the Story Advisor's Verdict. Criterion 5 reads the verdict structurally; if it reads FRACTURED with non-Phase-4 fractures, the remedy is to address the fracture upstream, not to override the Verdict here.
• Does not design ablation or propose new matrix cells. Those are Phase 4's and the Architect's jobs respectively.
• Does not compare across Phase 3 runs that pre-dated a Case B / Case C re-entry. The matrix of record is whatever the latest framework_v{m}.md declares; stale earlier-version runs are not evaluated here.

---

Phase 3 → Phase 4 routing — the Main Agent's reading

Once this skill returns PASS with Aggregate rollback depth None and a handoff-compatible story check, the Main Agent's job (outside this skill) is to:

1. Write PHASE_3_SUMMARY.md at phase_3_main_experiment/PHASE_3_SUMMARY.md — one page: framework version, cells covered, per-Core-Claim verdict snapshot, latest story check verdict, accepted deviations.
2. Invoke phase4-ablation (not yet built at time of writing) with inputs: framework_v{m}.md, the latest story_check, the matrix of analyses, and the Findings snapshot. Ablation design should use the story check's contribution bullets to pick attribution targets.
3. Record the transition in _logs/decision_log.md — Phase 3 exit is a governance moment.

For non-None depths, the routing entry points are the same ones used at Phase 2 exit (Case B of the upstream skill), with the invocation brief naming the Phase-3-specific analysis and cell that triggered the rollback. Resume Phase 3 as Case C after the upstream fix; do not restart the full matrix unless the upstream change was spec-wide (a new framework_v{m'}.md).

For the Phase 0 re-ideation path specifically: there is no new loop name — the entry point is loop1-ideation-pipeline Case C, same as from phase2-exit-check. The Critical severity at Phase 3 differs from Phase 2's only in evidence weight: a Phase 3 Critical has seen the full matrix fail, not just a slice.