audit-reproducibility

# Audit Reproducibility Compare numeric claims in a manuscript (point estimates, standard errors, p-values, counts) against the actual outputs produced by the analysis pipeline. Report PASS / FAIL per claim against the tolerance thresholds defined in [`.claude/rules/replication-protocol.md`](../../rules/replication-protocol.md). **Core principle:** If the paper says `ATT = -1.632 (0.584)` and the code produces `-1.628 (0.591)`, we verify — **numerically** — that the difference is within the documented tolerance. No more "looks close enough" eyeballing. **Two directions, not one.** *Vertically*, each claim is checked against the output that produced it. *Horizontally*, it is checked against every **other** artifact that displays the same number — the supplement table, the slide deck, the poster. The vertical check passes contentedly while a deck quotes last month's value; only the horizontal one catches that. Declared displays live in the passport's `appears_in` list — see [`replication-protocol.md` → The horizontal check](../../rules/replication-protocol.md#the-horizontal-check). ## When to use - **Before submission.** Catches the "I updated the analysis but forgot to update Table 2" bug. - **Before presenting or teaching from the same numbers.** Catches the deck, the poster, or the supplement that was never regenerated after the last rerun. - **Before releasing a replication package.** Verifies the code actually reproduces the paper. - **After a major revision.** Ensures the paper still matches the latest code. - **Quality-gate in `/commit`.** Pair with a pre-commit invocation on manuscript + analysis changes. ## Inputs - `$0` — path to the manuscript (`.tex`, `.qmd`, `.md`, `.pdf`). Required. - `$1` — path to the outputs directory. Defaults to `output/`, where every language's pipeline writes (R, Stata via [`/stata-replication`](../stata-replication/SKILL.md), Python). Recognised alternatives: `_targets/objects/` (R `targets` workflows), any directory the user-specified outputs live in. If `output/` does not exist but a pre-v2.6 `scripts/<lang>/_outputs/` does, use that and say so in the report. ## Workflow ### Phase 0: Pre-flight 1. Read [`replication-protocol.md`](../../rules/replication-protocol.md) for the tolerance thresholds currently in effect. 2. Verify the outputs directory exists and is non-empty. If empty or stale (older than the manuscript), prompt the user to re-run their pipeline (e.g., `Rscript scripts/R/00_run_all.R`) before auditing. 3. Ensure a `sessionInfo.txt` or equivalent environment capture exists in the outputs dir. ### Phase 1: Extract claims from the manuscript Parse the manuscript for numeric claims. Patterns to match: - **Point-estimate + SE**: `ATT = -1.632 (0.584)`, `$\beta = 0.342$ (0.091)`, `hat{\tau} = 1.28**` with starred significance - **Table cells**: `& -1.632$^{***}$ & 0.584 &` in LaTeX table environments - **Counts**: `our sample of 2,847 firms`, `$N = 2{,}847$` - **Summary stats**: `mean = 0.423`, `SD = 0.087` - **P-values**: `p < 0.01`, `$p = 0.003$` Record each claim as a tuple: ``` { claim_id: "Table2_col3_ATT", location: "Table 2, Column 3, row 'Treatment'", kind: "point_estimate" | "standard_error" | "p_value" | "count" | "percentage", reported_value: -1.632, uncertainty: 0.584, # only for point estimates significance_stars: 3, # 0-3 or None raw_context: "the ATT estimate of -1.632 (0.584) indicates..." } ``` Write the extracted claims to `quality_reports/reproducibility_claims_[manuscript-name].json` so the user can review the extraction before audit. ### Phase 2: Extract results from outputs Scan `$1` for corresponding values. Priority order: 1. **`.rds` files** — `readRDS(path)$coef[["treatment"]]` style lookups. Can use `Rscript -e "saveRDS(summary(readRDS(...)), '/tmp/audit.rds')"` to extract. 2. **`.tex` tables** — parse LaTeX table cells directly; match on column headers + row labels. 3. **`.csv` summary files** — pandas/readr parse, key-value lookup. 4. **`.out` / `.log` files** (Stata, regress output) — regex extraction. 5. **`.json`** — direct key lookup. Record each extracted result: ``` { source: "output/results.rds", lookup_key: "fit_main$coefficients['treated']", value: -1.628, uncertainty: 0.591, p_value: 0.005 } ``` ### Phase 2b: Collect every declared display (passport mode) For each passport claim, read its `appears_in` list and pull the value **as displayed** at each entry: ``` { claim_id: "C3", displays: [ { path: "manuscript.tex", locator: "Table 1, Col 2", display_precision: 3, shown: 0.342 }, { path: "Slides/Lecture04_Results.tex", locator: "frame 'Main result'", display_precision: 2, shown: 0.34 } ] } ``` Rules for this pass: - The primary `location:` is one of the displays, not a separate thing — if `appears_in` repeats it, that is one display, not two. - **A declared display that cannot be resolved — missing file, or no value at the `locator` — is recorded as `shown: NOT_FOUND`.** It resolves to FAIL in Phase 4c. Never skip it: a locator that has quietly stopped matching is exactly where a stale number hides. - A claim with **no** `appears_in` list is horizontally *unchecked*, not horizontally clean. Report it that way (see Phase 5) rather than silently passing it. ### Phase 3: Match claims to results Use fuzzy heuristics when exact labels don't match: - Name similarity (`"treatment effect"` ~ `"ATT"` ~ `"treated"`) - Magnitude similarity (if two candidates have values within 10% of the reported, prefer the one with closer SE) - Context hints from the claim's `raw_context` field (table number, row label, description) For every claim, produce a match candidate with a confidence score. Claims below 0.7 confidence get flagged as "UNMATCHED — manual review needed" rather than silently passing. ### Phase 4: Tolerance check For each matched claim, apply the thresholds from `replication-protocol.md`: | Kind | Tolerance | Example | |---|---|---| | Integers (N, counts) | Exact | 2,847 must equal 2,847 | | Point estimates | `abs(reported - computed)` < 0.01 | -1.632 vs -1.628 → diff = 0.004 → PASS | | Standard errors | `abs(reported - computed)` < 0.05 | 0.584 vs 0.591 → diff = 0.007 → PASS | | P-values | Same significance level | p<0.01 and p<0.01 → PASS; p<0.01 and p=0.03 → FAIL | | Percentages | ±0.1pp | 42.3% vs 42.35% → PASS | Respect any **tolerance overrides** the user has written into their `replication-protocol.md` fork (they may loosen for MC noise or tighten for administrative data). ### Phase 4b: Disposition — PASS / FAIL / EXPLAINED / UNMATCHED A tolerance check resolves to one of four dispositions: - **PASS** — within tolerance. - **FAIL** — outside tolerance, with no defensible alternative recorded. **Blocks** (exit 1). - **EXPLAINED** — outside tolerance, **but** the author has recorded a *concrete, named alternative specification* that accounts for the gap (see the downgrade rule). Surfaced in the report and carried into a response-to-referees; does **not** block. - **UNMATCHED** — no computed counterpart found (Phase 3 confidence < 0.7). Never auto-downgradable. **A mismatch is not automatically a failure.** In applied work the most common out-of-tolerance result is a *defensible alternative spec*, not a bug — `reghdfe` vs `feols` clustering df, a different bandwidth-selection rule, a different MC seed/reps, or display rounding. The skill's job is to *stage the disagreement* for a human auditor, not to pronounce the code right and the paper wrong. (The df-adjustment note in "Stata-specific notes" below is the canonical example of a named alternative.) **The manuscript is not the oracle.** When the computed value disagrees with the manuscript, do not presume the code is correct and the paper stale — nor the reverse. A refactor may have broken a previously-correct table (the *on-disk output* is the buggy one), or the paper may carry an old number. The computed value is a **challenger**, not ground truth. Report a mismatch as "one of {paper, code} must change — isolate which," never "revert the code to match the paper." This prevents the trap of reverting a genuine bug-fix just to make the paper 'reproduce.' #### Downgrade rule: FAIL → EXPLAINED A FAIL may be downgraded to EXPLAINED **only** when a *specific named alternative* is recorded for that exact claim — in the passport entry's `notes:` field (passport mode) or the audit report's author-note column (default mode). Example of a valid note: > "reghdfe vs feols clustering-df adjustment; under the reghdfe small-sample correction the published value is −1.19, within rounding of the script's −1.187. CODE-CORRECTED pending." The author is the **auditor**: the skill stages the two-sided comparison (reported value *and* computed value, both shown); the human writes the one-line named alternative; the skill records it and thereafter respects it. Tag the resolution `PAPER-CORRECTED`, `CODE-CORRECTED`, or `DEFENSIBLE-ALTERNATIVE`. **Hard floor — never downgradable to EXPLAINED:** - A blank note, "unclear", "looks fine", or any note that does not *name a concrete alternative spec*. - An **UNMATCHED** claim (no computed counterpart to compare against). - A flat numerical contradiction with no alternative offered. (Citation/existence claims are out of scope here — `/verify-claims` owns those, and applies the same named-alternative softening on its side.) #### Repeated EXPLAINED is a signal (two-strikes) Reuse the two-strikes rule from `review-paper --adversarial` and [`summary-parity.md`](../../rules/summary-parity.md): if the **same** claim is downgraded to EXPLAINED in **two consecutive audits** without ever being corrected to PASS (the author keeps invoking the alternative but never updates paper or code), stop treating it as quietly resolved. Surface it prominently in Phase 5 — *"this contested number has been EXPLAINED twice but never corrected"* — so a standing disagreement can't hide behind a recorded note indefinitely. In passport mode, detect this by comparing the current `status`/`notes` against the prior audit's. ### Phase 4c: Horizontal check — the displays must agree with each other Phase 4 compared the claim to the code. This phase compares the claim's displays to **each other**, pairwise over the `displays` collected in Phase 2b: 1. **Round both sides to the coarser precision.** Two displays agree when they are equal after both are rounded to the **smaller** of their two `display_precision` values. `0.34` in a deck against `0.342` in the paper agrees at 2 decimals; `0.29` against `0.342` does not. 2. **No tolerance applies.** The `tolerance:` block governs the vertical comparison, where two *measurements* are compared. Two displays of one number are two *copies* — after the rounding in step 1 they either match or they do not. 3. **Dispositions, in the vocabulary of Phase 4b:** - **PASS** — every pair agrees. - **FAIL** — any pair disagrees, or any display is `NOT_FOUND`. **Never downgradable to EXPLAINED**: a named alternative spec explains why code and paper differ and has nothing to say about why two copies of one number differ. If a display genuinely shows a *different* specification, it is a different claim and belongs in its own passport entry. - **UNCHECKED** — the claim declares no `appears_in` list. Reported, non-blocking; the fix is to declare the displays. 4. **Report which side moved, not which side is right.** Name the artifact whose value differs from the rest, and when it was last touched; the author decides whether the deck is stale or the paper is. Do not hand-edit one display to match its sibling — re-derive both from the output the claim points at ([`replication-protocol.md` → Anti-patterns](../../rules/replication-protocol.md#anti-patterns)). ### Phase 5: Report Write `quality_reports/reproducibility_audit_[manuscript-name].md`: ```markdown # Reproducibility Audit: [Manuscript Title] **Date:** [YYYY-MM-DD] **Manuscript:** [path] **Outputs directory:** [path] **Tolerance source:** .claude/rules/replication-protocol.md ## Summary | Status | Count | |---|---| | PASS (vertical and horizontal) | N | | FAIL (diff > tolerance, no named alternative) | M | | EXPLAINED (out of tolerance, named alternative recorded) | E | | UNMATCHED (manual review) | K | | HORIZONTAL DRIFT (declared displays disagree, or a display not found) | H | | HORIZONTALLY UNCHECKED (claim declares no `appears_in`) | U | | **Overall verdict** | **PASS / FAIL** (FAIL iff M > 0 or H > 0; EXPLAINED does not fail the audit) | ## PASS (all within tolerance) | Claim | Reported | Computed | Diff | Tolerance | |---|---|---|---|---| | Table2_col3_ATT | -1.632 (0.584) | -1.628 (0.591) | 0.004 / 0.007 | 0.01 / 0.05 | ## FAIL (outside tolerance — BLOCKER) | Claim | Reported | Computed | Diff | Tolerance | Location in paper | Author note (name a concrete alternative to downgrade → EXPLAINED) | |---|---|---|---|---|---|---| ## EXPLAINED (out of tolerance; defensible named alternative recorded — non-blocking, carry into response-to-referees) | Claim | Reported | Computed | Named alternative (why the gap is defensible) | Resolution | |---|---|---|---|---| | Table3_col2_ATT | -1.187 | -1.19 | reghdfe vs feols clustering-df adjustment | DEFENSIBLE-ALTERNATIVE | ## UNMATCHED (manual review) | Claim | Raw context | Candidate sources | |---|---|---| ## HORIZONTAL DRIFT (one number, two values — BLOCKER) | Claim | Display A | Display B | Compared at | Values | Which side moved | |---|---|---|---|---|---| | C3 | manuscript.tex — Table 1, Col 2 | Slides/Lecture04_Results.tex — frame 'Main result' | 2 decimals | 0.34 vs 0.29 | deck older than the claim's output_file | ## HORIZONTALLY UNCHECKED (no `appears_in` declared) | Claim | Primary display | Other artifacts to declare | |---|---|---| ## Environment [sessionInfo excerpt] ## Next steps 1. Resolve each FAIL row — either correct the manuscript, rerun the analysis, or (if the gap is a defensible alternative spec) record a concrete named alternative to downgrade it to EXPLAINED. 2. Resolve each HORIZONTAL DRIFT row — regenerate the lagging display from the claim's output, never by retyping the other display's value. A `NOT_FOUND` display is fixed by correcting the `locator`, not by dropping the entry. 3. Review UNMATCHED rows — add explicit lookup keys or widen the search scope. 4. Declare `appears_in` for the HORIZONTALLY UNCHECKED rows — every artifact a reader sees the number in. 5. Review EXPLAINED rows before submission — each should map to a sentence in the response-to-referees. 6. After zero FAILs and zero HORIZONTAL DRIFT (EXPLAINED rows allowed), the paper is replication-ready. ``` ### Phase 5b: Append to the replication log The report above and the passport are rewritten on every run, so neither keeps a history. The log does. After every run — default and passport mode alike — **append** one block to `quality_reports/replication-log.md`. It is committed, so a co-author or data editor can see what was checked, against which commit, and how each number was obtained, without reading the code. - **Append only.** Write with `>>`; never edit or reorder a past block. A correction is a new run, not an edit. The repo-hygiene gate fails a commit that edits or removes a committed line — at the pre-commit hook against the last commit, and in CI against the branch the work merges into. It proves no entry was edited, not that every run was logged. A disclosure redaction is the one exception: commit it with `ALLOW_LOG_REWRITE=1` and the reason in the commit message.