pipeline-audit

name

pipeline-audit

description

Audit a research data-code-output-manuscript pipeline for correctness and consistency. Use this skill whenever the user wants to verify their analysis pipeline, check for bugs in statistical code, find hardcoded numbers in a manuscript, or ensure narrative descriptions match the actual results. Trigger phrases include "check my pipeline", "audit my code", "make sure my results are consistent", "verify my manuscript", "find magic numbers", or any request to do a full review of a research project's code and writeup.

Research Pipeline Audit

This skill performs a comprehensive audit of a research project to ensure the data → code → output → manuscript pipeline is correct, clean, and internally consistent. It is designed to run with a capable model and catch errors a human reviewer might miss.

The Four Audit Tasks

Run all four in sequence. Report findings for each before moving to the next.

---

Task 1: Statistical Method Correctness

Goal: verify that the code actually implements what it claims to implement.

• Read all analysis scripts. Understand what each function does.
• For every statistical method used (DiD, TWFE, SDiD, regression, mixed models, meta-analysis, etc.), verify:
  • The model formula is correct for the stated estimand
  • Fixed effects, weights, and clustering are applied correctly
  • The outcome variable and treatment indicator are coded correctly
  • Pre/post and treatment/control splits are defined correctly
  • Any optimization problems (e.g. cvxpy) have correct objective, constraints, and variable shapes
  • Standard errors and confidence intervals are computed correctly
• Flag any discrepancy between what the code does and what the methods section says it does
• Check for off-by-one errors in indexing, period definitions, or subsetting
• Check that transformations (standardization, log, etc.) are applied consistently across train/test or pre/post splits

Report: list of confirmed-correct methods, and any bugs or discrepancies found with file + line number.

---

Task 2: Remove Dead Code

Goal: the codebase should contain only what is needed to reproduce the manuscript.

• Trace the full execution path from raw data to final manuscript figures, tables, and numbers
• Identify code that is never called, produces outputs not used in the manuscript, or is clearly exploratory (ad hoc prints, scratch computations, commented-out blocks that are not needed)
• Do NOT remove:
  • Robustness checks that are referenced in the paper (even if in supplementary)
  • Functions that are called indirectly
  • Any code the user has explicitly flagged as intentional
• Propose a clean list of what to remove, with justification for each item
• If a run_all.sh or equivalent entry point exists, verify it actually reproduces all outputs

Report: list of dead code with file + line numbers and a recommendation (remove / keep / ask user).

---

Task 3: Hunt for Magic Numbers

Goal: every number appearing in the manuscript should be generated by code, not typed by hand.

• Read the full manuscript (.tex, .Rmd, .qmd, or equivalent)
• Find every numeric value: effect sizes, sample sizes, percentages, p-values, confidence intervals, means, test statistics, dates, counts
• For each number, determine whether it is:
  • Generated: piped in from code output via \input{}, \Sexpr{}, params$, inline R, or equivalent → OK
  • Hardcoded: typed directly into the prose → FLAG
• Pay special attention to numbers in the abstract, results section, and figure/table captions — these are highest risk
• If the codebase uses a results file (e.g. results.tex, numbers.tex, output/stats.csv), verify the numbers in that file match what the analysis code produces

Report: list of hardcoded numbers with manuscript location (line number or paragraph), the value, and whether a corresponding code output exists that could replace it.

---

Task 4: Narrative–Results Consistency

Goal: what the paper says should match what the numbers show.

• For every quantitative claim in the manuscript, find the corresponding number in the output
• Flag mismatches such as:
  • Claiming significance when p > .05 (or whatever threshold is stated)
  • Claiming non-significance when p < threshold
  • Describing a direction of effect (e.g. "remote students performed worse") that contradicts the sign of the coefficient
  • Citing a specific value (e.g. "a 0.3 GPA decrease") that does not match the output
  • Describing group differences that are not statistically supported
  • Overstating effect size or precision
• Also check:
  • Figure descriptions match what the figure actually shows
  • Table notes and column headers match the model being reported
  • Footnotes citing specific numbers are accurate

Report: list of consistent claims (brief), and any mismatches with manuscript location + the actual value from the output.

---

How to Run This Audit

1. Ask the user for:

   • The root directory of the project (or confirm you can see it)
   • The main manuscript file path
   • The entry point script (e.g. run_all.sh) if one exists
   • Any files to exclude (e.g. old versions, archived code)

2. Start by building a map of the project:

   data/ → code/ → output/ → manuscript
   

   Understand which scripts produce which outputs, and which outputs are cited in the manuscript.

3. Run all four tasks. Be thorough — the value of this audit is catching things humans miss.

4. Produce a final report organized by task, with:

   • A summary verdict (clean / issues found)
   • Itemized findings with file paths and line numbers
   • Recommended actions (fix, remove, replace with code-generated value, etc.)

---

Output Format

## Pipeline Audit Report

### Task 1: Statistical Correctness
[PASS / ISSUES FOUND]
- ...

### Task 2: Dead Code
[CLEAN / ITEMS TO REMOVE]
- ...

### Task 3: Magic Numbers
[CLEAN / HARDCODED VALUES FOUND]
- ...

### Task 4: Narrative Consistency
[CONSISTENT / MISMATCHES FOUND]
- ...

### Summary
[Overall verdict and prioritized action list]

Keep findings actionable. For each issue, say exactly what the problem is, where it is, and what to do about it.