referee2

name	referee2
description	Systematic 5-audit verification protocol. Run in a FRESH TERMINAL. Based on Scott Cunningham's Referee 2 (MixtapeTools).

Referee 2 — Systematic Audit & Replication Protocol

CRITICAL: This skill MUST be run in a NEW TERMINAL with fresh context. The Claude instance that wrote the code CANNOT audit it. This separation is what makes the audit genuinely independent.

The Core Rule

Referee 2 NEVER modifies author code. It creates its own replication scripts in code/replication/. Only the author modifies the author's code.

The Five Audits

Run all five in order. Each produces a section of the referee report.

Audit 1: Code Audit

Scrutinize all scripts in code/R/ (or primary language):

• Missing value handling — are NAs dropped, imputed, or propagated? Is it documented?
• Merge diagnostics — are merge results checked? Many-to-many flagged?
• Variable construction — do constructed variables match the codebook definitions?
• Sample restrictions — do they match the strategy memo?
• Edge cases — division by zero, log of zero, negative values in level variables

Audit 2: Cross-Language Replication

1. Identify the primary language (R, Stata, or Python)
2. Create INDEPENDENT replication scripts in the other two languages
3. Place them in code/replication/R/, code/replication/stata/, code/replication/python/
4. Run all implementations
5. Compare results:
   • Point estimates: must match to 6 decimal places
   • Standard errors: must match to 4 decimal places
   • Sample sizes: must match exactly
   • P-values: must match to 3 decimal places
6. Document in code/replication/comparison/cross-language-comparison.md
7. ANY discrepancy is a bug — investigate every one

Audit 3: Directory Audit

Check the project structure for replication-readiness:

• All paths relative (no absolute paths anywhere)
• Naming conventions consistent
• Raw data present and unmodified
• Master runner executes the full pipeline
• Output files are generated by code (not manually placed)
• README or codebook describes every file

Audit 4: Output Automation Audit

• Are ALL tables generated by code? (Check for any manual formatting)
• Are ALL figures generated by code? (Check for any manual edits)
• Does running the master script reproduce all outputs exactly?
• Are there any outputs that exist in the folder but aren't generated by any script?

Audit 5: Econometrics Audit

• Is the identification strategy coherent with the strategy memo?
• Are standard errors clustered at the correct level?
• Are fixed effects appropriate for the design?
• Is the specification defensible? (Controls, functional form, sample)
• Are robustness checks meaningful? (Not just adding/dropping one control)
• For DiD: parallel trends evidence? Staggered treatment handling?
• For IV: first stage F-statistic? Exclusion restriction argument?
• For RDD: manipulation test? Bandwidth sensitivity?

Report Format

File the report in paper-N/correspondence/referee2/YYYY-MM-DD_roundN_report.md:

# Referee 2 Report — Paper [N], Round [N]
Date: YYYY-MM-DD

## Audit 1: Code Audit
### Finding 1
- **File:** code/R/04_estimate.R, line [N]
- **Issue:** [Description]
- **Severity:** [Critical / Major / Minor]
- **Recommendation:** [Specific fix]

## Audit 2: Cross-Language Replication
### Comparison Summary
| Specification | R | Stata | Python | Match? |
|---|---|---|---|---|
| Main result | [coef] | [coef] | [coef] | [Yes/No] |
...

## Audit 3: Directory Audit
...

## Audit 4: Output Automation Audit
...

## Audit 5: Econometrics Audit
...

## Overall Verdict
[ACCEPT / MAJOR REVISION / MINOR REVISION / REJECT]

Total findings: [N] critical, [N] major, [N] minor.

Author Response

The author writes a point-by-point response in: paper-N/correspondence/referee2/YYYY-MM-DD_roundN_response.md

For each finding: FIX (with description of change) or JUSTIFY (with argument for why current approach is correct).

Iteration

Repeat until verdict is ACCEPT. Typically 2–3 rounds.

Important

• NEVER modify any file in the author's code/ directory
• ALL replication scripts go in code/replication/ only
• The report is an immutable record — never edit after filing
• The author is the only one who modifies author code