phase6-exit-check

name

phase6-exit-check

description

Exit rubric for Phase 6 (Writing). Load after phase6-writing has produced phase_6_writing/drafts/paper_v{n}.md by assembling per-section drafts, and the Main Agent needs to determine whether Phase 6 is done (structurally complete) and what the routing signal is (Proceed to Phase 5 Case C for paper- draft review vs. revise specific sections vs. escalate to Phase 5 Case C anyway for semantic review). Returns PASS or NEEDS-REWORK with a routing- advisory table. Evaluates structural completeness of the paper draft and per-section files, claim-evidence trace fidelity, and anchor consistency (contribution bullets / spec modules / summary tables) — NOT the prose quality (that is Phase 5 Case C's job), NOT whether the contribution bullets are 'right' (that is Story Advisor's job), NOT whether the draft is submission- ready (that is Phase 5 Case C's Overall Recommendation). Does NOT re-draft any section, does NOT arbitrate between Writer choices and reviewer concerns, does NOT auto-route.

Phase 6 Exit Check

Read phase_6_writing/drafts/paper_v{n}.md for the latest {n}, the per- section files under phase_6_writing/sections/ pinned in the paper_v{n}.md header, the confirmed phase_3_main_experiment/specs/framework_v{m}.md, the latest phase_3_main_experiment/story_checks/story_check_{k}.md, phase_3_main_experiment/PHASE_3_SUMMARY.md, phase_4_ablation/PHASE_4_SUMMARY.md (if present), phase_5_review/PHASE_5_SUMMARY.md, both latest phase_5_review/reports/review_{constructive|adversarial}_v{n_rev}.md reports, and phase_0_ideation/ideas/IDEA.md. Apply the seven criteria below in order. Each criterion is binary: pass or fail.

A single failing criterion means the overall result is NEEDS-REWORK.

This check is structural and anchor-consistent. It asks: has Phase 6 produced a handoff-ready draft whose every empirical claim traces to a named source, whose abstract and introduction speak the same contribution bullets the story_check authorizes, whose method section describes the spec's modules, and whose results section reports the numbers the summaries contain? It does not re-judge prose quality (that is Phase 5 Case C's remit), does not re-evaluate contribution bullets (Story Advisor), does not decide whether the research is publishable (that is Phase 5's Overall Recommendation applied against the draft). The routing signal — which sections need revision, whether the draft is ready for Case C review, whether the claim-evidence traces need tightening — is surfaced for the Main Agent to act on; this check does not auto-route.

---

Criterion 1 — Paper draft exists and assembly is well-formed

Pass if phase_6_writing/drafts/paper_v{n}.md exists for the highest round index {n} in the drafts/ directory AND its header declares the §Sections pinned block with one entry per section in the declared section set AND each pinned sections/{section}_v{s}.md file actually exists on disk at the declared path.

Fail if the paper draft is missing, OR the §Sections pinned block is absent or malformed, OR any pinned section file does not exist, OR the pinned section order in the header does not match the final-paper order (abstract → introduction → related_work → method → experiments → results → discussion → conclusion, unless the invocation brief declared a custom section set).

To check: read the header, list the section paths, verify each exists.

---

Criterion 2 — Every section file is structurally complete

Pass if each pinned sections/{section}_v{s}.md satisfies the Writer's §Output Format:

• Header fields present: Section handle, Invocation case, Source framework spec, Source story check, Phase 3 summary, Phase 4 summary, Phase 5 summary, Reviewer reports read, Prior draft (or "none (Case A)"), Writer-level concerns scoped count, Date.
• §Drafting rubric block present — either with one entry per scoped concern (concern handle + severity + guidance + address location) OR with the explicit "No Writer-level concerns scope to this section" marker.
• §Section draft block present and non-empty. For sections that conventionally contain tables (results, experiments), tables are present (markdown tables, not placeholders).
• §Residual notes block present (may be short or list items or a single paragraph; may declare "None").
• §Revision notes block present if Case B (or explicitly "Not applicable (Case A)" if Case A).

Fail if any single sub-check fails on any section file. List which section and which sub-check.

---

Criterion 3 — Claim-evidence traces are dense and traceable

Pass if every paragraph containing an empirical claim in any section file carries at least one <!-- trace: {citation} --> comment, AND the citation points at one of the allowed traceable anchors:

• analysis_{n}.md §Claim-Result row X (Phase 2/3/4 analysis path)
• story_check_{k}.md §Contribution-Bullets §{handle}
• framework_v{m}.md §Modules §{module} or §Runner or §Claim-Trace
• PHASE_3_SUMMARY.md §Per-Claim-Verdict row X or §Main-Results §{cell}
• PHASE_4_SUMMARY.md §Per-Variant row "{handle}"
• IDEA.md §Known-Related-Work row X or §Motivation or §Core-Claims §{claim}
• PHASE_5_SUMMARY.md §Writer-level-concerns §{handle} (for reviewer-sourced framing citations)

AND the named artifact actually exists on disk (spot-check via file reads; need not verify every citation but should sample at least 5 random empirical-claim paragraphs per section type and confirm citations are not hallucinated).

"Empirical claim" includes: numbers, percentages, deltas, "outperforms" / "improves" / "reduces" / "matches" / "fails" language, named cells or matrix scopes, attribution claims ("component X drives the improvement"). Conceptual prose (motivation transitions, definitions, framing sentences) is exempt from trace requirements.

Fail if any section has paragraphs with empirical claims lacking traces, OR any trace citation names an artifact that does not exist (hallucinated citation), OR traces point at non-allowed anchor types (e.g., a raw prose quote from IDEA rather than a structured §Core- Claims reference).

This criterion enforces the Writer agent's §Core Principles #4: evidence-grounded prose with inline traces. Phase 6's load-bearing invariant is that every number in the paper can be traced back to the evidence that produced it.

---

Criterion 4 — Writer-level concern coverage

Pass if every Writer-level concern from both latest reviewer reports (extracted by Routing hint = "Writer-level framing") appears in at least one section file's §Drafting rubric block AND is either:

• Addressed in the §Section draft via an <!-- addresses: {concern handle} --> trace, OR
• Documented in §Residual notes with a specific reason it cannot be addressed in prose (typically because the fix requires upstream evidence).

Concerns are matched structurally: a concern handle like review_constructive_v2.md §Concerns §C3 should appear by that exact handle. The exit-check computes section-scope routing structurally — if PHASE_5_SUMMARY does not declare per-section scope for a concern, the exit-check accepts any section's rubric block as a valid address location AND does not fail the criterion merely because the section choice is ambiguous. What fails: the concern is in no rubric block anywhere AND is not in any §Residual notes block.

Fail if any Writer-level concern is absent from every section file's §Drafting rubric AND every §Residual notes block, OR any concern is listed as "addressed" in a §Drafting rubric block but the §Section draft contains no <!-- addresses: ... --> trace for it.

This criterion ensures the reviewer-surfaced drafting rubric has been actually read and acted on (or explicitly declined with reason), not silently ignored.

---

Criterion 5 — Contribution-bullet fidelity

Pass if the abstract, introduction (opening paragraphs with contribution statement), and conclusion use language consistent with the latest story_check_{k}.md's Contribution Bullets block:

• Every contribution claim in the abstract paraphrases a bullet that the latest story_check marks as SUPPORTED or with provisional-hedged framing. No bullet retired in the latest story_check appears as a contribution in the abstract, introduction, or conclusion.
• Provisional-marked bullets are hedged in prose (e.g., "our ablation evidence is consistent with ..." or "preliminary results suggest ..."), not stated outright.
• No contribution claim appears in these sections that is not backed by a story_check bullet (invented contributions are rejected).

The check is structural: sample each bullet handle from the latest story_check, search for its presence / absence in abstract + introduction + conclusion, and flag mismatches.

Fail if any bullet retired in the latest story_check appears as a contribution in abstract / introduction / conclusion, OR any provisional bullet is stated without hedging, OR the abstract / introduction / conclusion makes contribution claims with no matching bullet.

This criterion enforces the Writer's §Core Principles #3: contribution bullets are the claim surface, verbatim-in-attribution-state.

---

Criterion 6 — Spec-module and summary-table fidelity

Pass if the method section + results section + experiments section are consistent with their upstream anchors:

Method section sub-checks:

• Every spec §Module in the latest framework_v{m}.md (excluding the Runner/Driver if it is purely an orchestration module) has a corresponding subsection or paragraph in the method section with a trace to the module's §Modules §{module} anchor.
• No method section subsection describes a component not in the spec §Modules (invented components are rejected).
• The operation order described in method prose matches the spec's §Runner §Operation narrative (reading the prose in order should recover the spec's call order).

Experiments section sub-checks:

• Datasets named in the experiments section match the matrix scope in PHASE_3_SUMMARY.md §Matrix-Scope (or equivalent). Datasets mentioned in prose but not in the matrix scope are a fail.
• LLMs, baselines, and metrics named in the experiments section match the summary's declared set. Additions or omissions are a fail.
• Iteration strategy prose is consistent with the spec §Runner §Operation.

Results section sub-checks:

• Every row in the results section's main-results table appears in PHASE_3_SUMMARY.md §Main-Results (or equivalent). Added rows or invented cells are a fail.
• Every row in the results section's ablation table appears in PHASE_4_SUMMARY.md §Per-Variant (or equivalent). Added rows are a fail.
• Numerical values in result tables are consistent with the summary entries (spot-check at least 3 cells; the exit-check need not verify every cell but should catch gross discrepancies).

Fail if any sub-check fails. List the specific section + sub- check with the discrepancy.

This criterion enforces the Writer's §Core Principles #5 and #6: honor Analyst and Story Advisor verdicts, hedge at the evidence boundary. Drift between prose and the structured summaries is the most common failure mode of narrative drafting and must be caught here.

---

Criterion 7 — Matrix-scope claim calibration

Pass if every claim of generality in abstract + introduction + discussion is anchored to the matrix scope declared in PHASE_3_SUMMARY.md:

• Claims phrased as "across datasets" / "across settings" / "across model families" / "in general" are backed by matrix coverage that spans multiple datasets / LLMs / etc. A claim of "across datasets" with a matrix of one dataset is a fail.
• Claims narrowed to specific cells ("on HotpotQA with Llama-3-8B") are allowed without matrix-wide support.
• Single-cell supported Core Claims produce narrowed prose in the abstract / introduction, not generalized prose. If the abstract says "our method improves EM" and the matrix shows improvement on one of three datasets only, that is a fail (overclaim).

Fail if any section makes a generality claim not supported by matrix scope, OR any SUPPORTED-on-subset result is reported as matrix-wide in abstract / introduction.

This criterion is the prose-level analog of the Constructive / Adversarial Reviewer's Dimension 1 attacks. Catching overclaims here before Phase 5 Case C saves a review round.

---

Output

After evaluating all seven criteria, output exactly this block:

## Phase 6 Exit Check

**Result: PASS / NEEDS-REWORK**

| Criterion | Result | Note |
|-----------|--------|------|
| 1. Paper draft exists and assembly is well-formed | PASS / FAIL | [confirm paper_v{n}.md + pinned sections] |
| 2. Every section file is structurally complete | PASS / FAIL | [list any structural sub-check failures by section + field] |
| 3. Claim-evidence traces are dense and traceable | PASS / FAIL | [count trace-lacking paragraphs; flag hallucinated citations] |
| 4. Writer-level concern coverage | PASS / FAIL | [count concerns addressed / scoped / declined in residual notes; flag any unaddressed-and-not-declined] |
| 5. Contribution-bullet fidelity | PASS / FAIL | [flag retired-bullet leaks, un-hedged provisionals, invented contributions] |
| 6. Spec-module and summary-table fidelity | PASS / FAIL | [list drift items by section + sub-check] |
| 7. Matrix-scope claim calibration | PASS / FAIL | [flag overclaims by section + quote the offending sentence's location] |

**Failing criteria:** [list by number, or "none — all criteria passed"]

**Round:** {n} (paper_v{n}.md)
**Case:** A (fresh) | B (revision from v{n-1})
**Sections pinned this round:**

| Section | Pinned version | Drafted this round (A/B) | Concerns scoped | Concerns addressed | Residual notes count |
|---------|----------------|---------------------------|-----------------|--------------------|-----------------------|
| abstract | v{s} | Yes / No (carried forward) | {count} | {count} | {count} |
| introduction | v{s} | ... | ... | ... | ... |
| related_work | ... | ... | ... | ... | ... |
| method | ... | ... | ... | ... | ... |
| experiments | ... | ... | ... | ... | ... |
| results | ... | ... | ... | ... | ... |
| discussion | ... | ... | ... | ... | ... |
| conclusion | ... | ... | ... | ... | ... |

**Writer-level concern disposition (aggregate across sections):**

| Concern handle | Severity | Section scope | Addressed in draft? | Residual-note deferral? |
|----------------|----------|---------------|---------------------|-------------------------|
| review_constructive_v{n_rev}.md §Concerns §{c_handle} | Fatal/Major/Minor | {section} | Yes (trace: {addr}) / No | None / {reason} |
| review_adversarial_v{n_rev}.md §Objections §{o_handle} | Fatal/Major/Minor | {section} | Yes (trace: {addr}) / No | None / {reason} |

[One row per Writer-level-framing concern / objection from the latest reviewer reports.]

**Claim-evidence trace density (per section):**

| Section | Empirical-claim paragraphs | Traces present | Trace coverage | Hallucinated citations |
|---------|---------------------------|----------------|----------------|------------------------|
| abstract | {n} | {n} | {pct} | {count} |
| introduction | ... | ... | ... | ... |
| ... | ... | ... | ... | ... |

**Drift flags (anchor-consistency failures):**

[One entry per drift detected in Criterion 5 / 6 / 7. Empty if none.

- **{section}**: {drift type — e.g., "retired-bullet leak", "invented
  component in method", "ablation-table row not in PHASE_4_SUMMARY",
  "overclaim of dataset generality"}. Location: {paragraph / table row}.
  Anchor expected: {citation}. Suggested fix-section for Writer Case B:
  {section name}.]

**Routing signal for the Main Agent (advisory, NOT a decision):**

| Aggregate shape | Typical Main-Agent target |
|-----------------|---------------------------|
| All 7 criteria PASS | Proceed to Phase 5 Case C (paper-draft review). Main Agent writes PHASE_6_SUMMARY.md and re-invokes phase5-review with paper_v{n}.md as the Case C input. |
| Only Criterion 3 FAIL (trace density / hallucinated citations) in a subset of sections | Writer Case B on the flagged sections. Traces are a mechanical fix; re-enter phase6-writing Case B. |
| Only Criterion 2 FAIL (structural completeness) on a subset of sections | Writer Case B on the flagged sections. Structural issues are a mechanical fix. |
| Criterion 5 FAIL (contribution-bullet drift) | Writer Case B on abstract + introduction + conclusion at minimum. If drift is severe, also re-invoke Story Advisor to verify the bullet state is what Writer should be reflecting. |
| Criterion 6 FAIL (spec / summary drift) | Writer Case B on method / experiments / results. If multiple drifts suggest the Writer misread the spec or summary, flag to Main Agent for a PHASE_5_SUMMARY clarification pass before Writer Case B. |
| Criterion 7 FAIL (overclaim) | Writer Case B on abstract + introduction + discussion. Overclaims are a calibration fix; if pervasive, indicates the Writer-level concern coverage (Criterion 4) was superficial and the rubric needs sharpening. |
| Multiple criteria FAIL with >50% of sections flagged | Consider whether the draft should be discarded and a fresh Case A run attempted; surface to user. Continuing Case B repeatedly on a draft with pervasive drift is worse than starting over with a sharpened PHASE_5_SUMMARY. |
| All criteria PASS AND Phase 5 Case C has already run on paper_v{n-1}.md with "ready-to-write" recommendation AND paper_v{n}.md's section pins match v{n-1}'s | Proceed to submission-prep. Rare fast path — typically Phase 5 Case C needs a re-run on the new draft first. User-confirmed only. |

**Phase 5 Case C handoff readiness (advisory):**

- **Ready for Phase 5 Case C** if Criteria 1–7 all PASS. The draft is
  structurally sound and anchor-consistent; Case C will evaluate
  semantic framing, prose quality, and whether the reviewer-level
  concerns are actually addressed (beyond Writer's own self-report).
- **Not ready for Phase 5 Case C** if any criterion fails. Fix in
  Writer Case B first; Case C on a structurally incomplete draft
  wastes a review round.

**Claim-strength calibration risk (advisory):**

- If Criterion 7 PASSes but the paper has many hedged claims relative
  to SUPPORTED bullets, note it — this is not a failure, but Phase 5
  Case C may surface it as underclaiming (Writer-level concern,
  constructive side). Flag counts of hedged-vs-outright claims per
  section for Main Agent awareness.

**Next-step hint:**

- If PASS and routing = "Proceed to Phase 5 Case C": Main Agent writes
  `phase_6_writing/PHASE_6_SUMMARY.md` (rounds run, Phase 5 Case C
  cycles, final section pinnings, Writer-level-concern closure
  citations, residual-note deferrals), then re-invokes
  `phase5-review` as Case C with paper_v{n}.md path in the
  invocation brief.
- If PASS and routing = "Proceed to submission-prep": Main Agent
  executes submission-prep (outside framework scope — `[Cite: ...]`
  resolution, trace-stripping, LaTeX conversion).
- If NEEDS-REWORK: Main Agent re-enters `phase6-writing` as Case B
  with the flagged sections in the invocation brief. Do NOT proceed
  to Phase 5 Case C on a structurally incomplete draft.

---

What this skill does NOT do

• Does not re-judge prose quality. Sections may be clumsy, clunky, or narratively weak; Phase 5 Case C evaluates that. This check verifies the draft is structurally sound and anchor-consistent only.
• Does not re-evaluate contribution bullets. If the bullet set in the latest story_check is wrong, that is a Story Advisor problem surfaced upstream. This check verifies the Writer reflected the bullets accurately; it does not judge the bullets.
• Does not re-interpret Writer-level concerns. Concerns are matched structurally against section rubric blocks. Whether a concern was truly addressed in prose is a Phase 5 Case C judgment; this check confirms the Writer declared an address location or a deferral reason.
• Does not decide whether the draft is publishable. Phase 5 Case C's Overall Recommendation speaks to publishability applied to the draft. This check confirms the draft is ready for Case C review.
• Does not auto-route. Handoff-only. The Main Agent decides with the user whether to re-enter Phase 6 Case B, invoke Phase 5 Case C, or pursue submission-prep.
• Does not check submission-prep items. Bibliography resolution ([Cite: ...] → \cite{bibkey}), trace-stripping, LaTeX build success, formatting — all outside this check's scope.
• Does not compare across Phase 6 rounds. Only the latest paper_v{n}.md is evaluated. Prior rounds' drafts exist for Case B continuity only.
• Does not judge whether "enough" sections exist. The default section set is hard-coded into Criterion 1; custom section sets are honored as declared in the paper header. Arguing over section set is a phase6-writing Sub-process 1 decision, not this check's.
• Does not arbitrate between Writer and reviewer concerns. When the Writer declines to address a concern in §Residual notes with a reason, this check accepts the declination — Phase 5 Case C evaluates whether the decline is reasonable. This check only confirms the declination was declared, not silently omitted.

---

Phase 6 → Phase 5 Case C / Writer Case B routing — the Main Agent's reading

Once this skill returns PASS and the Routing signal table indicates "Proceed to Phase 5 Case C", the Main Agent's job (outside this skill) is to:

1. Write PHASE_6_SUMMARY.md at phase_6_writing/PHASE_6_SUMMARY.md — one page: rounds run, Phase 5 Case C cycles run so far (likely 0 on first handoff), final section pinnings, per-section concern-closure citations, deferrals in §Residual notes with reasons, outstanding Writer-level concerns the draft could not close.
2. Invoke phase5-review as Case C with paper_v{n}.md as the paper-draft input and the reviewer briefs updated to Case C. Reviewers produce concerns that either:
   • Return empty (0 Writer-level-framing concerns, 0 upstream-routed concerns) with Overall Recommendation "ready-to-write" → draft is submission-eligible; proceed to submission-prep (outside framework).
   • Return Writer-level concerns → re-enter phase6-writing as Case B with the affected sections.
   • Return upstream-routed concerns (Ablation / Phase 3 / Story Advisor / etc.) → Loop 4 → Phase 5 Case B → Phase 6 Case B cascade.
3. Record the transition in _logs/decision_log.md — Phase 6 exit is a governance moment (the first complete paper draft exists, and the research has its first prose-level surface).

For Writer Case B routings (this check fails on specific sections), the entry point is phase6-writing as Case B with the flagged section list in the invocation brief. The Main Agent should also verify whether the flagged drift suggests upstream fixes are needed:

• Criterion 5 FAIL (contribution-bullet drift) may indicate the Writer misread story_check_{k}.md. Re-read both and determine whether Writer correction or Story Advisor revision is the right call.
• Criterion 6 FAIL (spec / summary drift) may indicate the summaries are incomplete (PHASE_3_SUMMARY / PHASE_4_SUMMARY missing a row the results table references). Verify the summary sources before re-invoking Writer.
• Criterion 7 FAIL (overclaim) may indicate PHASE_5_SUMMARY's Writer-level concerns did not capture the overclaim risk effectively. Surface to user for whether PHASE_5_SUMMARY needs sharpening before Writer Case B.

After Writer Case B lands, re-run this exit-check. Cycle terminates when PASS and routing = "Proceed to Phase 5 Case C" is reached, and then again when Phase 5 Case C returns "ready-to-write" on the resulting draft.

For the halt path: if repeated Phase 6 Case B cycles fail to close drift AND Phase 5 Case C continues to surface Writer-level concerns that the Writer cannot address in prose, the deeper diagnosis is typically that the research's claim surface is not well-matched to its evidence base — route to Story Advisor revision with a direct brief (not via Writer concern routing), or in severe cases surface to user with Loop 4 / Phase 1 Case B / Phase 0 re-ideation as options.