phase5-exit-check

name

phase5-exit-check

description

Exit rubric for Phase 5 (Review). Load after both reviewer- constructive and reviewer-adversarial have produced reports for the current round, and the Main Agent needs to determine whether Phase 5 is done (structurally complete) and what the aggregated signal is (Proceed to Phase 6 vs. Loop 4 supplementary vs. halt). Returns PASS or NEEDS-REWORK with a routing-advisory table (Phase 6 vs. Loop-4-target vs. halt). Evaluates structural completeness of the two reviewer reports and their severity aggregation — NOT the correctness of individual concerns, NOT whether Loop 4 is 'really' needed (that is the Main Agent's call with the user), NOT paper- readiness (that is Phase 6's job). Does NOT re-review the research, does NOT arbitrate disagreements between reviewers, does NOT auto-route.

Phase 5 Exit Check

Read both phase_5_review/reports/review_constructive_v{n}.md and phase_5_review/reports/review_adversarial_v{n}.md for the latest {n}, the confirmed phase_3_main_experiment/specs/framework_v{m}.md, the latest phase_3_main_experiment/story_checks/story_check_{k}.md, phase_3_main_experiment/PHASE_3_SUMMARY.md, phase_4_ablation/PHASE_4_SUMMARY.md (if present), and phase_0_ideation/ideas/IDEA.md. Apply the six criteria below in order. Each criterion is binary: pass or fail.

A single failing criterion means the overall result is NEEDS-REWORK.

This check is structural. It asks: has Phase 5 produced a handoff-ready reviewer-gap report? It does not re-review the research, re-interpret individual reviewer concerns, arbitrate disagreements between the two reviewers, decide whether Loop 4 is "really" needed, or decide which Loop-4 target is correct. The routing signal — aggregated severity counts, routing-target distribution, reviewer agreement signal — is surfaced in the output for the Main Agent to act on with the user, but is not itself a criterion.

---

Criterion 1 — Both reviewer reports exist for the latest round

Pass if phase_5_review/reports/review_constructive_v{n}.md AND phase_5_review/reports/review_adversarial_v{n}.md both exist for the same {n} (the highest round index in the directory), and {n} is consistent between them.

Fail if one reviewer report is missing, OR the two reports have different {n}, OR the directory is empty (no reviewer reports for any round). This typically means phase5-review was interrupted between reviewer invocations — the Main Agent should re-invoke the missing reviewer on the current round.

To check: ls phase_5_review/reports/ and confirm the pairing.

---

Criterion 2 — Both reports are structurally complete

Pass if each report satisfies the structural requirements declared in its agent's §Output Format and §Quality Criteria sections:

Constructive report sub-checks:

• Header fields present: Invocation case, Source framework spec, Source story check, Phase 3 summary, Phase 4 summary (or absence noted), Paper draft (or absence noted), Prior review (or "none"), Analyses read, Date.
• Overall Recommendation present and one of the four allowed values (ready-to-write / minor-revision / major-revision / fundamental-rework).
• Reviewer's note section present (even if short).
• Research surface reconstruction present (non-empty).
• Strengths section has at least 2 entries with Evidence citations, OR §Residual notes explicitly flags thin strengths and explains.
• Concerns section has ≥1 entry OR explicitly states "No concerns."
• Every concern entry has all five required fields: Issue, Why-a- reviewer-would-care, Evidence citation (pointing at specific artifact), Smallest-intervention, Routing hint (from the allowed list).
• Resolved concerns section present if Case B (or explicitly "Not applicable (Case A/C)").

Adversarial report sub-checks:

• Header fields present (same list as Constructive).
• Summary verdict present (one-phrase plus 2–4-sentence assessment).
• Central claim restatement present, grounded in contribution- bullet language from the latest story_check.
• Lead attack section present — either a specific Fatal attack, or the explicit "No single lead attack — objections are cumulative" disclaimer.
• Objections section has ≥1 entry OR explicitly states "No objections."
• Every objection entry has all four required fields: Attack, Evidence citation, Why-severity, Routing hint (from the allowed list).
• Resolved objections section present if Case B (or explicitly "Not applicable (Case A/C)").

Fail if any single sub-check fails. List which report and which sub-check.

---

Criterion 3 — Every concern and objection has severity + routing hint

Pass if every concern in the Constructive report and every objection in the Adversarial report carries:

• A severity label in {Fatal, Major, Minor} (exactly one of these values per concern/objection).
• A routing hint from the allowed list (Ablation Designer Case B / Architect Case B / Phase 1 Case B / Experimental Setup revisit / Phase 3 Case B / Story Advisor revision / Refiner round / Writer-level framing / Scout re-invocation / Tester re-invocation / Unfixable within project scope — the last only in the Adversarial report).

Fail if any concern or objection has no severity, OR has multiple severities, OR has a severity outside the allowed set; OR has no routing hint, OR the routing hint is not in the allowed list, OR the routing hint is phrased as narrative ("the authors should add X") rather than as a named pipeline entry point.

This is a hard structural check — the exit-check's aggregate counts and routing advisory depend on every entry having a normalized severity and routing hint.

---

Criterion 4 — Severity is consistent with the Overall Recommendation / Summary Verdict

Pass if the Constructive Overall Recommendation is consistent with its concern severity distribution per its §Quality Criteria:

• ready-to-write: 0 Fatal, ≤2 Major
• minor-revision: 0 Fatal, 3–5 Major (OR ≤2 Fatal whose routing hints are all "Story Advisor revision")
• major-revision: 1 Fatal or >5 Major
• fundamental-rework: ≥2 Fatal or ≥5 Fatal+Major

AND the Adversarial Summary Verdict's one-phrase summary is consistent with its severity distribution:

• "publishable pending X" / "publishable at target venue" ⇔ 0 Fatal (or Fatal routing exclusively to Writer-level framing)
• "needs major work on Y" ⇔ 1–2 Fatal or ≥3 Major
• "fundamental gap on Z" ⇔ ≥3 Fatal
• "unpublishable without re-ideation" ⇔ ≥1 Fatal with routing hint "Unfixable within project scope" OR "Refiner round on IDEA"

Fail if either reviewer's stated verdict contradicts its own severity distribution. Example failure: a Constructive report labeled "ready-to-write" with 3 Fatal concerns; an Adversarial report with verdict "publishable pending X" but 5 Fatal objections. This indicates the reviewer's own output is internally inconsistent and the agent needs to be re-invoked with a corrective brief.

Note: this criterion does NOT check whether the two reviewers agree with each other. They are allowed to disagree — that is the signal Criterion 5 surfaces. It only checks each report is internally consistent.

---

Criterion 5 — Agreement signal is computed and surfaced

Pass if the exit-check's Output block (§Output below) includes:

• An Agreements table: concerns/objections from both reviewers that share the same Routing hint AND cite overlapping evidence (same contribution bullet, same analysis row, same module, same baseline absence). One row per agreement pair.
• A Disagreements table: concerns/objections present in one reviewer's report but not the other's (non-matching routing hint + evidence), and concerns/objections with severity mismatches on overlapping evidence (Constructive Minor + Adversarial Fatal on same bullet, etc.). One row per disagreement.
• A one-line interpretation of the agreement density (e.g., "High agreement: 4 of 5 concerns shared" OR "Low agreement: reviewers found mostly disjoint gaps, indicating broad review surface").

Fail if the Agreements / Disagreements computation is absent from the output, OR it is populated but the method is not structural (i.e., the exit-check tried to judge whether two concerns are "really the same" based on narrative similarity rather than matching by Routing hint + Evidence citation).

This criterion exists because agreement density is a first-order routing signal — high-agreement Fatal findings are very high priority for Loop 4; low-agreement Minor findings are low priority. Without this surfaced explicitly, the Main Agent would have to read both reports in full to triage, defeating the purpose of the exit-check.

---

Criterion 6 — Evidence citations are traceable

Pass if every concern's / objection's Evidence citation names a specific traceable artifact:

• An analysis row ("analysis_{n}.md §Claim-Result row X")
• A spec module ("framework_v{m}.md §Modules §{module}")
• A contribution bullet ("story_check_{k}.md §Contribution-Bullets §{handle}")
• A summary table entry ("PHASE_3_SUMMARY.md §Per-Claim-Verdict row X")
• A Known Related Work row ("IDEA.md §Known-Related-Work row X")
• An explicit absence ("no analysis measures Y", where Y is named)
• A paper draft section (Case C only, "phase_6_writing/ sections/{file}.md §section")

AND the named artifact actually exists on disk (spot-check via file reads; the exit-check need not verify every citation but should sample enough to confirm citations are not hallucinated).

Fail if any concern or objection has an Evidence field that is a narrative gesture ("the experiments seem thin" / "the ablation coverage is weak" / "a reviewer would find this unconvincing") without a specific traceable artifact, OR cites an artifact that does not exist (missing analysis number, non- existent module name, etc.).

This criterion enforces the reviewer agents' §Core Principles #4: evidence-grounded concerns only. A reviewer report full of vague gestures is not handoff-ready.

---

Output

After evaluating all six criteria, output exactly this block:

## Phase 5 Exit Check

**Result: PASS / NEEDS-REWORK**

| Criterion | Result | Note |
|-----------|--------|------|
| 1. Both reviewer reports exist for round {n} | PASS / FAIL | [confirm pairing; flag missing report] |
| 2. Both reports are structurally complete | PASS / FAIL | [list any sub-check failures by report + field] |
| 3. Every concern/objection has severity + routing hint | PASS / FAIL | [count entries; flag any missing label / hint] |
| 4. Severity consistent with Overall Recommendation / Summary Verdict | PASS / FAIL | [flag any internal inconsistency] |
| 5. Agreement signal computed and surfaced | PASS / FAIL | [describe Agreements/Disagreements tables below] |
| 6. Evidence citations are traceable | PASS / FAIL | [flag any narrative-only citations] |

**Failing criteria:** [list by number, or "none — all criteria passed"]

**Round:** {n}
**Constructive Overall Recommendation:** {verbatim}
**Adversarial Summary Verdict:** {verbatim one-phrase summary}

**Aggregate severity counts (across both reports):**

| Severity | Constructive | Adversarial | Union (deduped by routing + evidence) |
|----------|--------------|-------------|---------------------------------------|
| Fatal | {c_f} | {a_f} | {u_f} |
| Major | {c_mj} | {a_mj} | {u_mj} |
| Minor | {c_mn} | {a_mn} | {u_mn} |
| Total | {c_total} | {a_total} | {u_total} |

**Agreements (both reviewers surfaced the same gap):**

| Constructive handle | Adversarial handle | Shared routing hint | Shared evidence | Max severity |
|---------------------|--------------------|--------------------|-----------------|--------------|
| {c_handle} | {a_handle} | {hint} | {evidence citation} | {Fatal/Major/Minor} |

[One row per agreement pair. Or: "No agreements — reviewers found
disjoint gaps."]

**Disagreements and one-sided findings:**

| Source | Handle | Routing hint | Evidence | Severity | Notes |
|--------|--------|--------------|----------|----------|-------|
| Constructive only | {c_handle} | {hint} | {evidence} | {severity} | [one line] |
| Adversarial only  | {a_handle} | {hint} | {evidence} | {severity} | [one line] |
| Severity mismatch | {c_handle} / {a_handle} | {hint} | {evidence} | C={sev} / A={sev} | [one line] |

[One row per item. Or: "No disagreements."]

**Agreement interpretation:** [One line — e.g., "High agreement: 4 of
5 Fatal/Major concerns are shared across reviewers, concentrated on
{routing target}" OR "Low agreement: reviewers raised mostly
disjoint concerns, suggesting broad review surface rather than a
single fatal gap."]

**Routing-target distribution (union of routing hints, weighted by severity):**

| Routing target | Fatal | Major | Minor | Total |
|----------------|-------|-------|-------|-------|
| Ablation Designer Case B | ... | ... | ... | ... |
| Architect Case B / Phase 1 Case B | ... | ... | ... | ... |
| Experimental Setup revisit | ... | ... | ... | ... |
| Phase 3 Case B rerun | ... | ... | ... | ... |
| Story Advisor revision | ... | ... | ... | ... |
| Refiner round on IDEA | ... | ... | ... | ... |
| Writer-level framing | ... | ... | ... | ... |
| Scout / Tester re-invocation | ... | ... | ... | ... |
| Unfixable within project scope | ... | ... | ... | ... |
| Total | ... | ... | ... | ... |

**Routing signal for the Main Agent (advisory, NOT a decision):**

| Aggregate shape | Typical Main-Agent target |
|-----------------|---------------------------|
| 0 Fatal + 0–2 Major (all routing: Writer-level framing) | Proceed to Phase 6 (Writing). Writing can absorb framing tweaks; no Loop 4 needed. |
| 0 Fatal + 0–2 Major (routing mixes Writer-level + Story Advisor) | Proceed to Phase 6, with a Story Advisor revision pass as a preliminary step. |
| 0 Fatal + 3–5 Major (routing concentrated on Ablation Designer / Phase 3) | Loop 4 — supplementary evidence. Route per routing-target distribution. Re-enter phase5-review as Case B after. |
| 1 Fatal (any routing) | Loop 4 — targeted fix. Fatal findings block Phase 6 until closed. Route per Fatal's routing hint. Re-enter Case B. |
| 2+ Fatal (routing mixes upstream targets) | Loop 4 — multi-target. Route all Fatals; re-enter as Case B only after all Fatals close. Consider whether the union of targets suggests deeper rollback (Phase 1 Case B or Refiner round) rather than targeted fixes. |
| 2+ Fatal routing includes "Unfixable within project scope" | **Halt** — surface to user. This is a Phase-0-re-ideation signal or project-scope revision signal. User-confirmed routing only. |
| ≥1 Fatal routing to "Refiner round on IDEA" AND Constructive Overall = fundamental-rework AND Adversarial verdict = "unpublishable without re-ideation" | **Halt** — surface to user with Phase 0 re-ideation (`loop1-ideation-pipeline` Case C) as the primary option. User-confirmed only. |

**Loop 4 decision context (advisory):**

- **Loop 4 is typically needed** if any Fatal exists OR if ≥3 Major
  concerns route to the same upstream target (e.g., 3 Major
  concerns all routing to "Ablation Designer Case B" are worth a
  targeted Phase 4 Case B pass).
- **Loop 4 is typically not needed** if all Fatal/Major concerns
  route to Writer-level framing OR Story Advisor revision — those
  can be absorbed in Phase 6 / Story Advisor without rerunning
  experiments.
- **Loop 4 target selection**: use the Routing-target distribution
  table's total column — the target with the most severity-
  weighted mass is the primary Loop 4 target. Multiple Fatals
  across targets usually need sequential Loop 4 passes (not
  parallel), because each pass's output becomes the next pass's
  input.

The Main Agent makes the actual routing call, weighing user
input and the routing-target distribution. This section is
advisory only.

**Paper-draft detection (for Case C handling):**

- Paper draft present in reports' headers: {yes — phase_6_writing/
  exists | no — Case A or B}
- Writing-level concerns in reports: {count — these only route to
  Writer-level framing; signal for the Main Agent that a post-
  draft Phase 5 review has occurred}

**Next-step hint:**

- If PASS and routing = "Proceed to Phase 6": Main Agent writes
  `phase_5_review/PHASE_5_SUMMARY.md` (rounds run, Loop 4 cycles,
  final recommendation + verdict, gap-closure citations), then
  invokes Phase 6 (Writing). Phase 6's skill is NOT YET BUILT.
- If PASS and routing = "Loop 4": Main Agent routes per the
  per-target Loop-4 target table above, with user confirmation
  for Fatal routings and all multi-target routings. After Loop 4
  lands, re-enter `phase5-review` as Case B.
- If PASS and routing = "Halt": Main Agent surfaces to user; do
  not auto-route.
- If NEEDS-REWORK: Main Agent re-invokes the offending reviewer
  with a structural-correction brief (via `phase5-review` Case C
  or a targeted re-invocation). Do not proceed to Phase 6 or
  Loop 4 on a structurally incomplete report set.

---

What this skill does NOT do

• Does not re-review the research. Both reviewer reports are authoritative. Criteria 2–6 check structural completeness and internal consistency; they do not re-weight concerns or override severity.
• Does not arbitrate disagreements. When the two reviewers disagree (Constructive Minor vs. Adversarial Fatal on the same evidence), Criterion 5 surfaces the disagreement to the Main Agent as routing-uncertainty. It does not pick a winner.
• Does not decide Loop 4. The Routing signal table is advisory. The Main Agent decides with the user — especially for Fatal routings and multi-target routings where a deeper rollback (Phase 1 Case B / Refiner round / Phase 0 re-ideation) may be more economical than sequential Loop 4 passes.
• Does not auto-invoke Phase 6, Loop 4, Phase 0, or any upstream skill. Handoff-only. The Main Agent routes with user confirmation.
• Does not compare across Phase 5 rounds. The round of record is the latest {n}. Prior rounds' reports exist for Case B reviewer continuity; this exit-check evaluates only the latest round.
• Does not evaluate paper readiness. The Constructive Overall Recommendation and Adversarial Summary Verdict speak to reviewer-legible readiness; the exit-check's job is to confirm those verdicts are structurally well-formed and surface the aggregate signal. Paper-draft quality is Phase 6's concern.
• Does not judge whether the reviewers were "right". If a reviewer missed a concern the Main Agent thinks matters, the remedy is a user-directed re-invocation (Case B with a corrective brief), not an override here.

---

Phase 5 → Phase 6 / Loop 4 routing — the Main Agent's reading

Once this skill returns PASS and the Routing signal table indicates "Proceed to Phase 6", the Main Agent's job (outside this skill) is to:

1. Write PHASE_5_SUMMARY.md at phase_5_review/PHASE_5_SUMMARY.md — one page: rounds run, Loop 4 cycles completed, final Constructive Overall Recommendation, final Adversarial Summary Verdict, final aggregate severity counts, Loop-4-closed concerns (with citations to the evidence that closed them), accepted-open concerns (concerns the Main Agent chose not to close — e.g., Minors accepted into Phase 6 as writing-level adjustments).
2. Invoke phase6-writing (not yet built at time of writing) with inputs: framework_v{m}.md, latest story_check, PHASE_3_SUMMARY.md, PHASE_4_SUMMARY.md, PHASE_5_SUMMARY.md, and both latest reviewer reports. Writing should treat Writer-level concerns as the drafting rubric.
3. Record the transition in _logs/decision_log.md — Phase 5 exit is a governance moment (the research's reviewer- legibility is locked).

For Loop 4 routings (Fatal present, or ≥3 Major concentrated on one upstream target), the entry points are:

• Ablation Designer Case B: phase4-ablation Case B with the specific new variant(s) named from the concerns' intervention clauses.
• Phase 3 Case B rerun: phase3-main-experiment Case B scoped to the additional cells named.
• Experimental Setup revisit: experiment-setup-pipeline then phase3-main-experiment Case B.
• Architect Case B / Phase 1 Case B: phase1-method-design Case B scoped to the component extension.
• Story Advisor revision: re-invoke story-coherence-advisor with a revision brief (Context C preferred if Phase 4 evidence supports the reframe; Context B if the reframe is pre- ablation).
• Refiner round on IDEA: idea-developer with the Fatal concern's framing as the direction brief.
• Scout / Tester re-invocation: literature-scout with the novelty-gap brief, or idea-pressure-tester with the attack- angle brief from the Adversarial Reviewer.

After Loop 4 lands, re-enter phase5-review as Case B. The cycle terminates when PASS + "Proceed to Phase 6" is reached or when the Main Agent + user determine the research is not salvageable within scope (halt path).

For the halt path specifically: the entry point is loop1-ideation-pipeline Case C with user-confirmed Phase 0 re-ideation, OR explicit project abandonment (not a pipeline entry — a user-level decision).