| name | referee2 |
| description | Systematic 5-audit verification protocol. Run in a FRESH TERMINAL. Based on Scott Cunningham's Referee 2 (MixtapeTools). |
Referee 2 — Systematic Audit & Replication Protocol
CRITICAL: This skill MUST be run in a NEW TERMINAL with fresh context.
The Claude instance that wrote the code CANNOT audit it.
This separation is what makes the audit genuinely independent.
The Core Rule
Referee 2 NEVER modifies author code. It creates its own replication scripts in code/replication/. Only the author modifies the author's code.
The Five Audits
Run all five in order. Each produces a section of the referee report.
Audit 1: Code Audit
Scrutinize all scripts in code/R/ (or primary language):
- Missing value handling — are NAs dropped, imputed, or propagated? Is it documented?
- Merge diagnostics — are merge results checked? Many-to-many flagged?
- Variable construction — do constructed variables match the codebook definitions?
- Sample restrictions — do they match the strategy memo?
- Edge cases — division by zero, log of zero, negative values in level variables
Audit 2: Cross-Language Replication
- Identify the primary language (R, Stata, or Python)
- Create INDEPENDENT replication scripts in the other two languages
- Place them in
code/replication/R/, code/replication/stata/, code/replication/python/
- Run all implementations
- Compare results:
- Point estimates: must match to 6 decimal places
- Standard errors: must match to 4 decimal places
- Sample sizes: must match exactly
- P-values: must match to 3 decimal places
- Document in
code/replication/comparison/cross-language-comparison.md
- ANY discrepancy is a bug — investigate every one
Audit 3: Directory Audit
Check the project structure for replication-readiness:
- All paths relative (no absolute paths anywhere)
- Naming conventions consistent
- Raw data present and unmodified
- Master runner executes the full pipeline
- Output files are generated by code (not manually placed)
- README or codebook describes every file
Audit 4: Output Automation Audit
- Are ALL tables generated by code? (Check for any manual formatting)
- Are ALL figures generated by code? (Check for any manual edits)
- Does running the master script reproduce all outputs exactly?
- Are there any outputs that exist in the folder but aren't generated by any script?
Audit 5: Econometrics Audit
- Is the identification strategy coherent with the strategy memo?
- Are standard errors clustered at the correct level?
- Are fixed effects appropriate for the design?
- Is the specification defensible? (Controls, functional form, sample)
- Are robustness checks meaningful? (Not just adding/dropping one control)
- For DiD: parallel trends evidence? Staggered treatment handling?
- For IV: first stage F-statistic? Exclusion restriction argument?
- For RDD: manipulation test? Bandwidth sensitivity?
Report Format
File the report in paper-N/correspondence/referee2/YYYY-MM-DD_roundN_report.md:
# Referee 2 Report — Paper [N], Round [N]
Date: YYYY-MM-DD
## Audit 1: Code Audit
### Finding 1
- **File:** code/R/04_estimate.R, line [N]
- **Issue:** [Description]
- **Severity:** [Critical / Major / Minor]
- **Recommendation:** [Specific fix]
## Audit 2: Cross-Language Replication
### Comparison Summary
| Specification | R | Stata | Python | Match? |
|---|---|---|---|---|
| Main result | [coef] | [coef] | [coef] | [Yes/No] |
...
## Audit 3: Directory Audit
...
## Audit 4: Output Automation Audit
...
## Audit 5: Econometrics Audit
...
## Overall Verdict
[ACCEPT / MAJOR REVISION / MINOR REVISION / REJECT]
Total findings: [N] critical, [N] major, [N] minor.
Author Response
The author writes a point-by-point response in:
paper-N/correspondence/referee2/YYYY-MM-DD_roundN_response.md
For each finding: FIX (with description of change) or JUSTIFY (with argument for why current approach is correct).
Iteration
Repeat until verdict is ACCEPT. Typically 2–3 rounds.
Important
- NEVER modify any file in the author's code/ directory
- ALL replication scripts go in code/replication/ only
- The report is an immutable record — never edit after filing
- The author is the only one who modifies author code
