review-paper

name	review-paper
description	Bootstrap a full paper review using katz
allowed-tools	Read, Bash, Glob, Grep
user-invocable	true

Review Paper

Orchestrates a full paper review using katz. Start here.

Usage

/review-paper

Workflow

Run katz guide overview to understand what katz can do and how to use it.

Then follow the typical workflow. At each step, check prerequisites before proceeding — if a step is already done, skip it.

1. Register the paper

Check: katz paper status — if "valid": true, skip to step 2.

Otherwise: katz guide skill register-paper — convert the PDF to markdown and register it.

2. Chunk into sections

Check: katz paper status — if "sections" > 0, skip to step 3.

Quick path: katz paper auto-chunk — automatically detects markdown headings and creates sections. Verify with katz paper status and spot-check a few sections with katz paper section <id>.

Manual path: katz guide skill chunk-paper — for more control over section boundaries.

3. Configure spotters

Check: katz spotter list — if spotters are already enabled, skip to step 4.

Follow: katz guide skill configure-spotters — read the paper, enable relevant spotters from the catalog, and add custom ones for paper-specific concerns.

4. Evaluate the paper

Check: katz eval list — if criteria are already enabled, proceed. Otherwise:

katz eval init-catalog

Then enable the criteria you want (or enable all):

# Enable specific ones
katz eval enable abstract_conveys_findings
katz eval enable design_matches_claims
# ... etc

Follow: katz guide skill eval-paper — read the paper and write narrative responses for each criterion.

5. Find issues

Run: katz guide skill edsl-find-issues — runs the EDSL-parallelized issue finder.

By default, uses 2 models (Claude Opus + GPT-5.4) and katz-enabled spotters.

• Pass --models 3 to add Gemini.
• Pass --builtin-spotters to use the 5 built-in spotters instead of katz-enabled ones.
• The script automatically deduplicates near-identical issues after filing.

6. Merge duplicate issues

The EDSL sweep often produces many near-duplicates (e.g., 5 issues about the same claim from different models). Before investigating, merge them:

katz issue merge --ids <id1>,<id2>,<id3> --title "Concise merged title"

This creates a single parent issue and marks the children as wontfix. Read through katz issue list --state draft and merge issues that point to the same underlying concern.

7. Investigate issues

Follow: katz guide skill investigate-issues — review each draft issue against the manuscript. Expect ~5–10% confirmation rate.

For each issue, read the manuscript context, determine a verdict (confirmed/rejected/uncertain), and record it with katz issue investigate and katz issue update.

8. Generate issue report

Run the report generator for the detailed issue-level HTML report:

katz report generate

Then open .katz/review.html to see the full report with issue cards, investigation verdicts, and manuscript quotes.

9. Write referee report

Follow: katz guide skill referee-report — synthesize the investigated issues into a narrative referee report.

This produces .katz/referee_report.md — a structured, professional review suitable for sharing with authors or an editor.

At each step, read the skill instructions and follow them. Use katz guide script <path> to inspect any scripts before running them.

End-to-end timing

A typical review of a 30-page paper with 14 spotters takes:

Step	Time
Register + chunk	2–5 min (PDF conversion is the bottleneck)
Configure spotters	2–3 min
Evaluate (eval-paper)	3–5 min
EDSL sweep	5–10 min (parallelized across EDSL remote runner)
Investigation	5–10 min (batch script approach)
Report + referee report	2–3 min
Total	~20–35 min

(Optional) Creating a GitHub issue

After the review, you can create a single GitHub issue summarizing all confirmed issues. Use gh issue create with a markdown body that includes:

1. Pipeline summary — how many calls, candidates, confirmed/rejected/uncertain
2. Confirmed issues grouped by category — e.g., "Identification & Design", "Statistical Interpretation", "Methodology", "Presentation"
3. Each issue as a checkbox (- [ ]) with:
   • Title and location (section, line number)
   • One-paragraph description of the problem
   • A blockquote of the relevant manuscript text
   • A concrete suggested fix
4. Footer noting the tool and commit hash

Building the issue body

Fetch full details for confirmed issues:

katz issue list --state confirmed | python3 -c "
import sys, json, subprocess
for i in json.load(sys.stdin):
    full = json.loads(subprocess.run(
        ['katz','issue','show',i['id']], capture_output=True, text=True
    ).stdout)
    print(json.dumps(full))
"

Then compose the markdown body using the issue titles, investigation notes, and resolved text. Group related issues into categories. Use a HEREDOC with gh issue create:

gh issue create \
  --title "Review: N confirmed issues from multi-model sweep" \
  --body "$(cat <<'EOF'
## Paper Review — Automated Issue Sweep

... markdown body ...

*Generated by [katz](https://github.com/expectedparrot/katz) on YYYY-MM-DD. Review commit: `abcdef12`.*
EOF
)"

Category suggestions for grouping

These categories work well for empirical economics papers:

• Identification & Experimental Design — SUTVA, selection, randomization concerns
• Statistical Interpretation — null-as-no-effect, multiple testing, power
• Methodology — collider bias, bad controls, conditioning on post-treatment
• Presentation — contradictory notes, math errors, generalizability of claims

Adapt categories to the paper's content.