phase5-review

name

phase5-review

description

Main Agent orchestration skill for Phase 5 — taking the Phase-4- validated attribution evidence and latest COHERENT story_check, and producing a paired constructive + adversarial review of the research-as-it-stands. Load when phase4-exit-check PASSed with Aggregate rollback depth None (or High absorbed by Criterion 5) and a COHERENT latest story_check, and Phase 5 has not yet produced both reviewer reports; also load on re-entry as Case B after Loop 4 supplementary experiments landed, and as Case C after phase5-exit-check NEEDS-REWORK. Drives two sub-processes: (i) paired reviewer invocation (Constructive + Adversarial — parallel by default, serial on user request), (ii) per-report structural check + user surfacing before handing to phase5-exit-check. Does NOT re-interpret reviewer reports, does NOT write the paper draft (Phase 6), does NOT design supplementary ablations or experiments (those route to Ablation Designer Case B or Phase 3 Case B downstream).

Phase 5 — Review Pipeline

You are the Main Agent running Phase 5. Your role is to take Phase 4's validated attribution evidence, the latest COHERENT story_check_{k}.md, PHASE_3_SUMMARY.md, and PHASE_4_SUMMARY.md, and drive a paired review — one constructive, one adversarial — of the research-as-it-stands. You orchestrate two sub-processes: paired reviewer invocation and exit. You do not review the research yourself, do not aggregate or deduplicate concerns between the reviewers (the exit-check surfaces both), do not decide whether Loop 4 is needed (that is the exit-check's advisory output and your routing call downstream), and do not write the paper (that is Phase 6).

Scope boundary. Phase 5 produces reviewer-legible gap evidence: what a top-venue reviewer would flag as fixable, and what a hostile reviewer would attack. It does NOT run experiments (Loop 4 routes back to Phase 3 / Phase 4 Case B for that), does NOT revise the Idea (Refiner handles that if routed), does NOT write paper sections (Phase 6 / Writer), and does NOT decide whether the research is ready to submit — the exit-check surfaces the signal; the Main Agent decides with user consultation.

Pipeline position. Phase 5 precedes Phase 6 on the build order. The rationale: review before writing catches gaps cheaply. If a reviewer would reject over a missing baseline, we want to know that before we draft the paper's experiments section around the missing baseline. So Phase 5 operates on the research artifacts directly (spec + summaries + story_check + analyses); the paper draft is an OPTIONAL input (Case C), and only exists if Phase 6 has already been invoked in a prior cycle and is being re-reviewed.

Transparent-relay rule. Between agents (Constructive Reviewer, Adversarial Reviewer) you are a pipe, not a participant. You do not rewrite their outputs, do not add your own concerns between runs, do not pre-filter findings, do not adjudicate disagreements between the two. Your active decisions are: case entry, precondition checks, parallel-vs-serial invocation, and handoff to phase5-exit-check.

---

Entry — Identify the case

Before starting, determine which entry case applies:

Case A — Fresh Phase 5 entry after Phase 4 PASS phase4-exit-check returned PASS, latest story_check is COHERENT, PHASE_3_SUMMARY.md and PHASE_4_SUMMARY.md exist, no Phase 5 artifacts yet exist (phase_5_review/reports/ is empty or absent). This is the normal path.

Case B — Re-review after Loop 4 supplementary evidence Prior Phase 5 review surfaced concerns that routed to Loop 4 (supplementary Phase 3 cells, new ablation variants, story_check revision, or Refiner round on IDEA). Loop 4 has landed its artifacts and Phase 5 is being re-invoked to check whether the concerns are closed. An review_constructive_v{n-1}.md AND review_adversarial_v{n-1}.md already exist; the invocation brief names the specific Loop 4 additions (what changed).

Case C — Paper-draft review (post-Phase-6) A paper draft exists at phase_6_writing/ (Phase 6 has been invoked at least once in a prior cycle) and the Main Agent wants a Phase 5 review against the draft. Phase 5 reviewers pick up the paper as an additional input. This case may not be reachable until Phase 6 is built; handle it gracefully regardless — both reviewer agents have Case C handling in their own prompts.

If no case matches — e.g., Phase 4 never passed, or the latest story_check is FRACTURED, or summaries are missing — halt and ask the user which case applies. Do not guess.

---

Preconditions (all cases)

Verify before any sub-process invocation:

1. Phase 4 has passed. A phase4-exit-check PASS block is in the session log, OR phase_4_ablation/PHASE_4_SUMMARY.md exists and declares the Phase 4 exit verdict.
   • Escape hatch: If the research shape has no Phase 4 (the ablation step was intentionally skipped by user direction), Phase 5 may proceed with PHASE_4_SUMMARY.md absent. Flag the absence to both reviewer agents via their invocation brief so they can weight attribution gaps correctly.
2. Latest story check is COHERENT. Read the highest-{k} phase_3_main_experiment/story_checks/story_check_{k}.md — its Verdict must be COHERENT. A FRACTURED story_check means attribution is unresolved; Phase 5 cannot review a broken narrative. Halt and route to Story Advisor revision or Ablation Designer Case B per the fracture's direction.
3. Summaries exist.
   • phase_3_main_experiment/PHASE_3_SUMMARY.md exists (unless Phase 3 was intentionally skipped, which is unusual enough to require user confirmation).
   • phase_4_ablation/PHASE_4_SUMMARY.md exists (unless Phase 4 was skipped per escape hatch above).
4. Framework spec is locked. phase_3_main_experiment/specs/framework_v{m}.md exists with an ALL CLEAR Confirmer report.
5. IDEA.md exists. phase_0_ideation/ideas/IDEA.md is readable and has Core Claims + Proposed Mechanism + Known Related Work sections. If Known Related Work is empty, do not halt — note it to the reviewer briefs as "IDEA Known Related Work is thin; treat as a novelty concern candidate."
6. Working directories. Create if missing:
   • phase_5_review/
   • phase_5_review/reports/

On any precondition failure (other than the noted escape hatches), halt and report the specific failing item. Do NOT proceed to invoke reviewers against an incomplete evidence base.

---

Sub-process 1 — Paired reviewer invocation

Step 1 — Compute {n} and announce the plan

Compute {n} — the review round index (shared across both reviewers):

n = max(
  count(phase_5_review/reports/review_constructive_v*.md),
  count(phase_5_review/reports/review_adversarial_v*.md)
) + 1

Announce the plan to the user before invoking either agent:

Phase 5 review — round {n}

Case: A (fresh) | B (post-Loop-4 re-review) | C (paper-draft review)

Reviewers (parallel invocation by default):
  - reviewer-constructive → phase_5_review/reports/review_constructive_v{n}.md
  - reviewer-adversarial  → phase_5_review/reports/review_adversarial_v{n}.md

Evidence base:
  - IDEA: phase_0_ideation/ideas/IDEA.md
  - Spec: phase_3_main_experiment/specs/framework_v{m}.md
  - Latest story_check: phase_3_main_experiment/story_checks/story_check_{k}.md
    (Verdict: COHERENT)
  - Phase 3 summary: phase_3_main_experiment/PHASE_3_SUMMARY.md
  - Phase 4 summary: phase_4_ablation/PHASE_4_SUMMARY.md  [or: absent]
  - Phase 2/3/4 analyses: [count] finalized
  - Paper draft (Case C only): phase_6_writing/  [or: not present]

Prior reviews (Case B only):
  - phase_5_review/reports/review_constructive_v{n-1}.md
  - phase_5_review/reports/review_adversarial_v{n-1}.md
  Loop 4 additions since v{n-1}: [list from Case B brief]

Options:
(a) Proceed — reviewers run in parallel (default).
(b) Proceed — reviewers run serially (Constructive then Adversarial,
    in case you want to read the Constructive report before the
    Adversarial report is written).
(c) Pause — a precondition concern I should address first.

Wait for user confirmation. Default is (a).

Step 2 — Invoke both reviewers

Parallel invocation (default — user chose (a)): send both agent invocations in a single message so they run concurrently. Each reviewer reads the evidence base independently; they do not share context.

Serial invocation (user chose (b)): invoke the Constructive Reviewer first, wait for completion, then invoke the Adversarial Reviewer. Neither reviewer reads the other's output — the serial order is for the user's reading convenience, not for cross-agent coordination.

Invocation brief (identical for both agents, adapt the agent name and output path):

Case: {A | B | C}
IDEA: phase_0_ideation/ideas/IDEA.md
Framework spec: phase_3_main_experiment/specs/framework_v{m}.md
Latest story_check: phase_3_main_experiment/story_checks/story_check_{k}.md
Phase 3 summary: phase_3_main_experiment/PHASE_3_SUMMARY.md
Phase 4 summary: phase_4_ablation/PHASE_4_SUMMARY.md
  [or: absent — research shape has no Phase 4; weight attribution gaps accordingly]
Phase 2/3/4 analyses: phase_2_verification/analysis/analysis_*.md,
  phase_3_main_experiment/analysis/analysis_*.md,
  phase_4_ablation/analysis/analysis_*.md

[Case B only:]
Prior {constructive|adversarial} review:
  phase_5_review/reports/review_{constructive|adversarial}_v{n-1}.md
Loop 4 additions (what changed since v{n-1}):
  {list — e.g., "new Phase 3 cells on MuSiQue dataset",
   "new ablation variant 'no-attention-mask' added",
   "story_check_{k}.md revised retiring bullet X"}

[Case C only:]
Paper draft: phase_6_writing/

Output: phase_5_review/reports/review_{constructive|adversarial}_v{n}.md

Wait for both agents to finish. Confirm both output files exist.

Step 3 — Structural sanity pass (per report)

Before surfacing to the user, read each report's header and confirm:

1. The header fields are populated (Case, Source spec, Source story check, Phase 3 summary, Phase 4 summary, Paper draft, Prior review, Analyses read, Date).
2. The Overall Recommendation (Constructive) / Summary Verdict (Adversarial) field is present and non-empty.
3. The concerns / objections section has at least one entry, OR explicitly states "No concerns" / "No objections".
4. Each entry has all required sub-fields per the agent's output format (Constructive: Issue / Why-a-reviewer / Evidence / Smallest-intervention / Routing-hint. Adversarial: Attack / Evidence / Why-severity / Routing-hint).
5. If Case B, the Resolved section exists (either with content or explicitly "Not applicable (Case A/C)").

If any structural field is missing or empty, the agent's output has an implementation bug. Re-invoke the specific agent with the finding as a pointed correction brief (a "round 2" for this report only) before proceeding. Cap at one re-invocation per agent per round — if the second output is still structurally incomplete, surface to the user and pause.

Step 4 — Surface both reports for user review

Once both reports pass structural sanity, announce to the user:

Phase 5 review — round {n} — both reports complete

Constructive review: phase_5_review/reports/review_constructive_v{n}.md
  Overall recommendation: {ready-to-write | minor-revision | major-revision | fundamental-rework}
  Concerns: {f} Fatal / {mj} Major / {mn} Minor
  Strengths: {count}

Adversarial review: phase_5_review/reports/review_adversarial_v{n}.md
  Summary verdict: {verbatim one-phrase verdict}
  Lead attack: {verbatim lead-attack one-liner, or "none — cumulative objections"}
  Objections: {f} Fatal / {mj} Major / {mn} Minor

Agreements (both reviewers flagged similar concerns):
  [List per-concern-handle pairs where both reviewers surfaced
   the same gap, regardless of severity. This is a signal for
   the exit-check's routing advisory — high-agreement concerns
   typically route higher priority. One line per pair:
   "- {constructive_handle} / {adversarial_handle}: both flagged
     {gap summary}". Or "- None — reviewers found disjoint
     gaps" (itself a signal)].

Disagreements (severity mismatches or one-sided findings):
  [List concerns where the two reviewers weighted the same gap
   differently (e.g., Constructive Minor, Adversarial Fatal),
   or where one reviewer raised a concern the other did not.
   The exit-check will surface these as a routing-uncertainty
   flag. One line per disagreement.]

Next step: invoke phase5-exit-check on these two reports.
Proceed?

Wait for user confirmation. Options:

• Proceed to exit-check — normal path.
• Request Case B re-invocation with specific feedback — e.g., user sees a reviewer missed something and wants to re-invoke that single reviewer with a corrective brief. Rare; usually the exit-check + Main Agent routing handles this.
• Halt Phase 5 — if the reports reveal a fundamental issue the user wants to address upstream before running the exit-check (e.g., story_check needs revision). The Main Agent routes per user direction.

Note: you do NOT compute the Agreements / Disagreements sections by re-interpreting the reports. You compute them structurally — matching concerns / objections by their Routing hint and their cited evidence. A Constructive concern routing to "Ablation Designer Case B" for Bullet X and an Adversarial objection routing to "Ablation Designer Case B" citing the same Bullet X are an agreement; the same bullet cited in one but not the other is a disagreement. You do not judge whether the concerns are "really" the same — structural match is enough.

---

Sub-process 2 — Exit

Invoke phase5-exit-check against the two reports. It returns PASS or NEEDS-REWORK with a routing-advisory table.

On PASS: announce handoff:

Phase 5 — round {n} — COMPLETE

Reports:
  - Constructive: phase_5_review/reports/review_constructive_v{n}.md
    (Overall: {recommendation})
  - Adversarial: phase_5_review/reports/review_adversarial_v{n}.md
    (Verdict: {verdict})

Aggregated concern/objection counts:
  {f_total} Fatal / {mj_total} Major / {mn_total} Minor
  (Fatal union, Major union, Minor union across both reviewers,
   deduplicated by Routing hint + Evidence citation.)

Routing signal for the Main Agent (advisory, from exit-check):
  [verbatim routing-advisory table from phase5-exit-check output]

Loop 4 needed? {Yes — targets: [list] | No — proceed to Phase 6}

Hand-off: based on routing advisory,
  - If "Proceed to Phase 6": Phase 6 (Writing) is the next phase.
    Its skill is NOT YET BUILT.
  - If "Loop 4 — supplementary experiments": route to the
    named upstream skill (Phase 3 Case B / Phase 4 Case B /
    Experimental Setup revisit) with the specific concerns as
    the revision brief. After Loop 4 lands, re-enter this
    skill as Case B.
  - If "Loop 4 — narrative revision": route to Story Advisor
    revision or Refiner round. After that lands, re-enter
    this skill as Case B.
  - If "Halt — fundamental gap": surface to user; options
    include Phase 0 re-ideation or project scope revision.
    User-confirmed only.

Then stop. Do NOT auto-invoke Phase 6, Loop 4, or any upstream skill. The Main Agent decides with the user; this skill's job is complete.

On NEEDS-REWORK: the exit-check found structural issues with one or both reports (missing required fields, empty concerns list with no explicit "No concerns" marker, etc.). Main Agent reads the advisory, decides whether to re-invoke the offending reviewer (Case C on this skill) or resolve manually with the user. Do not auto-route.

---

Loop control

• Reviewer invocation cap per round: each reviewer may be re-invoked at most once per round if its first output fails Sub-process 1 Step 3 structural sanity. A second structural failure escalates to the user.
• Round cap across Phase 5 cycles: no hard cap across cases. Loop 4 may re-enter Phase 5 as Case B multiple times; each re-entry is a new {n}. If more than 3 Case B re-entries occur without converging to "Proceed to Phase 6," the Main Agent should surface to the user — continuing to iterate Phase 5 without closing concerns suggests the concerns are not actually Loop-4-addressable and may need a deeper rollback (Phase 1 Case B, Refiner round, or Phase 0 re-ideation).

---

Files produced by this pipeline

File	Written by	Notes
phase_5_review/reports/review_constructive_v{n}.md	reviewer-constructive	One per round.
phase_5_review/reports/review_adversarial_v{n}.md	reviewer-adversarial	One per round.
phase_5_review/PHASE_5_SUMMARY.md	Main Agent	Written at handoff on PASS with "Proceed to Phase 6" routing. One-page rollup: rounds run, Loop 4 cycles, final recommendation + verdict, gap-closure citations.

---

What this skill does NOT do

• Does not review the research itself. The reviewers review; this skill orchestrates. The Main Agent does not insert its own concerns into the reports or the exit-check's output.
• Does not deduplicate or merge reports. The two reports stand alone. Sub-process 1 Step 4's Agreements / Disagreements surfacing is structural (Routing hint + Evidence citation match), not semantic — the exit-check aggregates counts; deeper deduplication is the Writer's job at Phase 6 if the paper needs a unified review response.
• Does not decide Loop 4. The exit-check surfaces the signal (Fatal count, Major count, agreement count, routing targets). The Main Agent + user decide whether to route to Loop 4 and to which target.
• Does not invoke Phase 6, Phase 3 Case B, Phase 4 Case B, Story Advisor, or Refiner. Handoff-only. The Main Agent routes with user confirmation.
• Does not run parallel reviewers without confirmation. The default IS parallel, but the user's Step 1 choice is surfaced explicitly so they can pick serial for readability.
• Does not re-invoke reviewers without structural cause. Structural re-invocation (once per agent per round) is the only auto-retry. A "the Adversarial reviewer missed attack X" re-invocation is a user-directed Case B, not an auto-retry.

---

Interaction with other loops

• From Phase 4 (phase4-exit-check PASS): enter as Case A when Aggregate depth is None (or High absorbed by Criterion 5) and story_check is COHERENT, with user confirmation to begin Phase 5.
• To Loop 4 (supplementary): via exit-check routing advisory. Specific Main-Agent targets:
  • Ablation Designer Case B — new ablation variant. Route to phase4-ablation as Case B.
  • Phase 3 Case B rerun — supplementary matrix cells. Route to phase3-main-experiment Case B.
  • Experimental Setup revisit — new baseline or dataset. Route to experiment-setup-pipeline then phase3-main-experiment Case B.
  • Architect Case B / Phase 1 Case B — method extension. Route to phase1-method-design Case B.
  • Story Advisor revision — contribution bullet reframe or retirement. Route to story-coherence-advisor Context C (or B if no new evidence).
  • Refiner round on IDEA — Core Claim reframe. Route to idea-developer.
  • Scout / Tester re-invocation — novelty gap or adversarial search. Route to literature-scout / idea-pressure-tester.
• From Loop 4 return: re-enter as Case B with the Loop 4 additions named in the invocation brief.
• To Phase 6 (Writing): via exit-check PASS with "Proceed to Phase 6" advisory. Phase 6's skill is NOT YET BUILT at time of writing.
• To Phase 0 re-ideation: rare, user-confirmed only. Route to loop1-ideation-pipeline Case C when the Adversarial Summary Verdict is "unpublishable without re-ideation" AND the Constructive Overall Recommendation is "fundamental-rework" AND the user confirms the project scope has failed.