research-review-loop

# Research Review Loop ## Quick start 1. Identify the target artifact, stakes, intended audience, and requested evidence class. 2. Initialize review state with `scripts/init_review_loop.py` unless a review pack already exists. Use `--from-paper-review <review_dir-or-final_issues.json>` when a first-pass paper-review bundle is available. 3. Build a claim ledger and a predecessor-failure ledger before issuing conclusions. 4. Review for internal consistency, evidence quality, external verifiability, provenance, non-vacuity, complete outcome accounting, and real rather than nominal independence. 5. Preserve previous rounds and append or version new outputs instead of overwriting prior review artifacts. 6. Update the latest `REVIEW_STATE.json`, `AUTO_REVIEW.md`, and `NARRATIVE_REPORT.md` after each round only after preserving prior versions. 7. If a `research-paper-review` bundle exists, import its issue IDs, quotes, ratings, and summary path rather than re-summarizing or downgrading the first-pass critique. 8. Infer the provenance of existing review artifacts before reorganizing them: upstream Claude/OpenAIReview bundles, Codex-native paper-review bundles, and review-loop hybrids are all valid inputs. 9. If the user asks for project-level sequencing, current-state inspection, or coordination across multiple research stages, invoke `research-pipeline-planner` first instead of treating the review loop as the whole task. 10. Use `../research-pipeline-planner/references/epistemic-assurance-contract.md` whenever a round is expected to authorize evidence promotion or route advancement. ## Relationship to sibling skills - `research-paper-review` should usually run before this skill when the artifact is a paper that has not yet received a first-pass critique. - `research-review-loop` owns tracked iterative review: issue carry-forward, resolution checks, accepted risks, and round discipline. - `research-results-auditor` owns detailed validity checks over concrete result artifacts; invoke it when a route decision turns on experiment outputs. - `research-rebuttal` owns responses to external reviewer comments and venue-constrained discussion artifacts. - `adversarial-doc-review` is broader and lighter-weight; use it for one-off document critique without a tracked research review state. ## Input contract - Minimum: - one concrete artifact under review - Prefer: - an existing review pack or tracked issue state - upstream Claude/OpenAIReview or Codex-adapted paper-review artifacts such as `summary.md`, `final_issues.json`, `review_summary.json`, `overall_assessment.txt`, `metadata.json`, `comments/`, `context/`, and `sections/` - revision diffs or an explicit statement of what changed since the last round - experiment plans, result artifacts, decision logs, failure reviews, and provenance records - current and requested evidence class plus the claimed independence arrangement ## Output contract - For future research-review-loop runs: preserve previous rounds and append/version new outputs instead of overwriting prior review artifacts. - Preserve prior rounds. Before writing a new round, copy, archive, or write into a round-specific directory so earlier `REVIEW_STATE.json`, `AUTO_REVIEW.md`, and `NARRATIVE_REPORT.md` artifacts remain available. - Primary tracked artifacts: - `REVIEW_STATE.json` - `AUTO_REVIEW.md` - `NARRATIVE_REPORT.md` - If upstream `paper-review` artifacts exist, keep explicit references to their file paths in the round state rather than rewriting the whole first-pass critique from scratch. - Preserve `impact_rating`, `confidence_rating`, `severity`, `quote`, `source_section`, and `related_sections` from `paper-review/final_issues.json` when importing first-pass paper-review issues. Map `impact_rating >= 4` to major-tracked issues, `impact_rating == 3` to moderate tracked issues, and `impact_rating <= 2` to minor tracked issues; if only upstream `severity` is present, preserve it as the tracked severity and leave numeric ratings null. - If the first-pass bundle lacks a root `review_summary.json` but has `metadata.json.round_summaries` or `round-N/review_summary.json`, use the latest round summary as the numeric/currentness summary and record the exact path in `source_artifacts`. - For any round that authorizes advancement, record the bounded verdict, current evidence class, requested evidence class, actual independence dimensions, unresolved predecessor failures, and the exact evidence supporting each closure. ## Hard stops - Do not authorize confirmatory or high-stakes advancement solely because all required files exist, validators pass, or the artifacts agree internally. - Do not mark a material assurance issue resolved by renaming a role, adding a field, changing a flag, replacing an artifact, or reclassifying the route without new evidence. - Do not call a review independent when the same effective process controls context, hidden data, implementation, evaluation, and advancement authority. - If a prior round identified hidden-truth leakage, vacuous comparisons, incomplete loss accounting, omitted failures, or self-attested controls, carry the issue forward until evidence resolves it or the claim is explicitly narrowed. - Continue useful exploratory review when stronger assurance is unavailable, but do not silently promote the evidence class. ## Workflow ### 1) Treat review as an iterative stateful process - Carry unresolved, resolved, and accepted issues across rounds. - Never collapse multiple review rounds into one untracked summary. - Never overwrite earlier round artifacts without first preserving them in a versioned or timestamped location. - Require each issue to have severity or impact rating, confidence, status, evidence, and a concrete fix or follow-up. - If `paper-review/final_issues.json` exists, initialize the first tracked issue set from that file instead of inventing a new initial ledger. Keep the paper-review issue title, quote, explanation, source section, and ratings traceable in the state. - Before deciding where to write the next round, inspect `metadata.json`, `artifact-index.md`, and existing `round-N/` folders. Continue the latest round number instead of flattening root paper-review files into a round folder. - Record which agent, model, prompt/context, code path, data access, and authority produced each stage-critical artifact when that provenance is available. ### 2) Build the claim and failure ledgers first - Extract definitions, assumptions, factual claims, quantitative claims, causal claims, speculative claims, control claims, and route-authorization claims. - Keep each claim traceable to a file location, section, figure, table, result artifact, or decision log. - Distinguish unsupported from false, and ambiguous from misleading. - For every material predecessor failure, record: - original failure and evidence - affected claim or route - current status - new evidence, accepted-risk rationale, or improper reclassification - Reclassification, replacement, or omission is not resolution. ### 3) Review in three passes - Internal pass: - consistency - terminology - logic - methodology - claim-to-evidence alignment - Assurance pass: - evidence class and selection history - complete decision or loss contract - non-vacuity and discriminating cases - hidden-information controls - successes, errors, omissions, skips, nulls, retries, timeouts, and initial failures - property-versus-label checks for locks, isolation, replay, role independence, and validators - actual independence across context, data, implementation, evaluation, and advancement authority - External pass: - time-sensitive facts - citations - benchmark claims - standards and rules - If external verification is unavailable, mark items as unverified and say what evidence would resolve them. ### 4) Force closure discipline - Do not mark a major issue resolved without new evidence, a revised artifact, or an explicit accepted risk. - Treat “future work” as a resolution only when the claim has been narrowed accordingly. - Do not resolve an imported paper-review issue by paraphrase alone; cite the revision diff, new analysis, narrowed claim, or accepted-risk rationale that changes its status. - Do not resolve an assurance issue because a self-attested field now says the desired property holds; inspect the mechanism and evidence. - An accepted risk must identify the unresolved failure, justify acceptance, narrow the affected claim or route, and remain visible in later rounds. - Prefer fewer high-signal issues over long undifferentiated lists. ### 5) Authorize advancement conservatively - Separate structural validity, internal consistency, exploratory usefulness, confirmatory support, and independent verification. - Require `research-results-auditor` when advancement depends on concrete experimental outputs that have not received a validity audit. - State the strongest remaining objection before the route decision. - Use `proceed`, `revise`, `narrow evidence class`, or `stop`; never use an unqualified “passed” when the bounded meaning matters. - If the runtime cannot provide material independence, record self-review and limit the verdict accordingly. ### 6) Keep the report actionable - Use `references/report-template.md` for each round. - Separate major issues from minor issues and open questions. - Include targeted rewrites when wording is the real problem. - Record what changed since the prior round and why each formerly-open issue is now resolved, deferred, accepted, narrowed, or still open. - List the minimum evidence needed to reach the next stronger evidence class. ## References - `references/review-checklist.md` - `references/report-template.md` - `references/review-state-schema.md` - `references/tabmol-ddi-ood-adapter.md` - `../research-pipeline-planner/references/review-stage-contract.md` - `../research-pipeline-planner/references/epistemic-assurance-contract.md` ## Script - `scripts/init_review_loop.py`: create deterministic state and report scaffolds for repeated review rounds; optionally seed open issues from `research-paper-review` output with `--from-paper-review`.