| name | jop-data-analysis |
| description | Use for analysis-stage decisions on a The Journal of Politics (JOP) manuscript — uncertainty, robustness, and reporting norms — written so the work is reproducible from line one. JOP makes acceptance contingent on replicability and a JOP replication analyst re-runs the code, so every reported number must come from a script. Guides analysis; it does not fabricate results. |
Data Analysis (jop-data-analysis)
At JOP, analysis and reproducibility are the same task: acceptance is contingent on
replicability, and a JOP replication analyst re-runs your code at conditional acceptance. Write the
analysis so that every number in the paper is regenerated by a script — and reported with honest
uncertainty within the page budget.
When to trigger
- Setting up the estimation/analysis pipeline
- Deciding which robustness checks belong in the main text vs the Online Appendix
- A reviewer asked for additional specifications, uncertainty, or sensitivity
- Preparing numbers that must match the deposited replication package exactly
Analysis norms
- Report uncertainty, not just point estimates: CIs, SEs (clustered appropriately), and
substantive effect sizes a general reader can interpret.
- Specification transparency: show the primary specification clearly; relegate the grid of
alternatives to the Online Appendix, but reference it.
- Robustness that targets the threat: each check should answer a specific objection (confounding,
functional form, sample, measurement), not pad the count.
- Multiple comparisons: adjust or pre-specify when testing many implications.
- Substantive interpretation: translate coefficients into quantities of interest (predicted
probabilities, marginal effects) — general-interest readers want magnitudes, not just stars.
Reproducible-from-line-one (the JOP analyst will re-run this)
- One master script runs everything in order and sets the working directory once.
- Set a seed for every stochastic step (bootstrap, simulation, MCMC, jitter, sampling).
- Record software and package versions for the readme (e.g., "R 4.3.1", "Stata/MP 18.0").
- Build a codebook naming and defining every variable used in the analysis.
- Tables and figures are generated by code, never hand-edited — numbers in the text must match.
Fit the analysis to the page budget
- Lead with the result that carries the argument; do not narrate every regression.
- Move the robustness grid, balance tables, and diagnostics to the Online Appendix (≤ 25 pp).
- A Short Article (≤ 10 pp) should show one clean, decisive analysis, not a buffet.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. Journal of Politics spans observational and experimental political science; report the identifying assumption and the magnitude, not just stars.
- Many outcomes / specifications:
romano_wolf (step-down FWER) or
benjamini_hochberg — report the adjusted threshold.
- OVB sensitivity:
oster_delta / sensemakr.
- Inference:
wild_cluster_bootstrap (few clusters), twoway_cluster / conley;
multilevel data → cluster at the right level.
- Re-fit off one handle:
audit_result(result_id) lists the missing checks and the
exact suggest_function for each.
- Exhibits:
etable / did_summary_to_latex from the handle — no retyped numbers.
Keep the decisive checks in the body and the exhaustive battery in the supplement. See
the executed chain in the JF execution walkthrough.
Anti-patterns
- Numbers in the manuscript that the deposited code cannot reproduce (fails the analyst check)
- Unseeded randomness or unpinned versions ("works on my machine")
- Star-gazing with no effect sizes or uncertainty a general reader can use
- Robustness checks chosen to inflate the count rather than rebut a threat
- Cramming every specification into the main text and blowing the page budget
What a JOP analysis referee is looking for
The reviewer pool spans subfields, so an analysis only a specialist can audit reads as fragile. Map each
demand to the move that satisfies it before the page count forces an ugly cut.
| Referee demand | Pass move | Fail signal |
|---|
| Usable magnitude | Marginal effect or predicted probability with CI | Coefficient stars, no magnitude in prose |
| Correct uncertainty | Cluster at assignment level; randomization inference | Default SEs on clustered or experimental data |
| Targeted robustness | Each check named to the threat it rebuts | A grid with no mapping to objections |
| Multiplicity honesty | Pre-specified families; adjusted p-values | One mined "significant" interaction |
| Reproducibility | Master script regenerates every number | "Available on request"; drifting numbers |
Worked micro-example (illustrative figures)
A hypothetical Short Article asks whether a state's adoption of automatic voter registration (AVR) raised
turnout, using a staggered difference-in-differences across states. The first pass runs naive two-way
fixed effects and reports a +3.1-point effect (illustrative). Because adoption is staggered, already-treated
states act as forbidden controls and the estimate carries negative-weight comparisons. The JOP-credible
re-analysis uses a heterogeneity-robust estimator (Callaway–Sant'Anna or Sun–Abraham), reports the
group-time average as +1.8 points, 95% CI [0.4, 3.2] (illustrative), shows flat pre-trends, and clusters
by state. The robustness grid goes to the Online Appendix, cited in one line of main text.
Referee pushback patterns and the JOP fix
- "Your DID uses naive TWFE on staggered adoption." Re-estimate with a heterogeneity-robust estimator,
show the event-study plot, and decompose the two-way estimate so the negative-weight problem is resolved.
- "Standard errors do not reflect the design." Cluster at the assignment level — the state in the AVR
example — with wild-cluster bootstrap when states are few.
- "This interaction looks fished." Show the pre-registered family and the adjusted p-value; concede a
null openly.
Output format
【Primary result】estimand + magnitude + uncertainty
【Robustness】each check ↔ the threat it answers (main vs appendix)
【Reproducible】master script + seeds + pinned versions + codebook? [Y/N]
【Numbers match】text == deposited output? [Y/N]
【Page discipline】main text lean, overflow in appendix? [Y/N]
【Next】jop-tables-figures
Supplementary resources
