Jop Data Analysis

Data Analysis (jop-data-analysis)

At JOP, analysis and reproducibility are the same task: acceptance is contingent on replicability, and a JOP replication analyst re-runs your code at conditional acceptance. Write the analysis so that every number in the paper is regenerated by a script — and reported with honest uncertainty within the page budget.

When to trigger

• Setting up the estimation/analysis pipeline
• Deciding which robustness checks belong in the main text vs the Online Appendix
• A reviewer asked for additional specifications, uncertainty, or sensitivity
• Preparing numbers that must match the deposited replication package exactly

Analysis norms

• Report uncertainty, not just point estimates: CIs, SEs (clustered appropriately), and substantive effect sizes a general reader can interpret.
• Specification transparency: show the primary specification clearly; relegate the grid of alternatives to the Online Appendix, but reference it.
• Robustness that targets the threat: each check should answer a specific objection (confounding, functional form, sample, measurement), not pad the count.
• Multiple comparisons: adjust or pre-specify when testing many implications.
• Substantive interpretation: translate coefficients into quantities of interest (predicted probabilities, marginal effects) — general-interest readers want magnitudes, not just stars.

Reproducible-from-line-one (the JOP analyst will re-run this)

• One master script runs everything in order and sets the working directory once.
• Set a seed for every stochastic step (bootstrap, simulation, MCMC, jitter, sampling).
• Record software and package versions for the readme (e.g., "R 4.3.1", "Stata/MP 18.0").
• Build a codebook naming and defining every variable used in the analysis.
• Tables and figures are generated by code, never hand-edited — numbers in the text must match.

Fit the analysis to the page budget

• Lead with the result that carries the argument; do not narrate every regression.
• Move the robustness grid, balance tables, and diagnostics to the Online Appendix (≤ 25 pp).
• A Short Article (≤ 10 pp) should show one clean, decisive analysis, not a buffet.

Execution bridge (StatsPAI / Stata MCP)

Run the battery, don't just enumerate it. Full map: execution-with-mcp. Journal of Politics spans observational and experimental political science; report the identifying assumption and the magnitude, not just stars.

• Many outcomes / specifications: romano_wolf (step-down FWER) or benjamini_hochberg — report the adjusted threshold.
• OVB sensitivity: oster_delta / sensemakr.
• Inference: wild_cluster_bootstrap (few clusters), twoway_cluster / conley; multilevel data → cluster at the right level.
• Re-fit off one handle: audit_result(result_id) lists the missing checks and the exact suggest_function for each.
• Exhibits: etable / did_summary_to_latex from the handle — no retyped numbers.

Keep the decisive checks in the body and the exhaustive battery in the supplement. See the executed chain in the JF execution walkthrough.

Anti-patterns

• Numbers in the manuscript that the deposited code cannot reproduce (fails the analyst check)
• Unseeded randomness or unpinned versions ("works on my machine")
• Star-gazing with no effect sizes or uncertainty a general reader can use
• Robustness checks chosen to inflate the count rather than rebut a threat
• Cramming every specification into the main text and blowing the page budget

What a JOP analysis referee is looking for

The reviewer pool spans subfields, so an analysis only a specialist can audit reads as fragile. Map each demand to the move that satisfies it before the page count forces an ugly cut.

Worked micro-example (illustrative figures)

A hypothetical Short Article asks whether a state's adoption of automatic voter registration (AVR) raised turnout, using a staggered difference-in-differences across states. The first pass runs naive two-way fixed effects and reports a +3.1-point effect (illustrative). Because adoption is staggered, already-treated states act as forbidden controls and the estimate carries negative-weight comparisons. The JOP-credible re-analysis uses a heterogeneity-robust estimator (Callaway–Sant'Anna or Sun–Abraham), reports the group-time average as +1.8 points, 95% CI [0.4, 3.2] (illustrative), shows flat pre-trends, and clusters by state. The robustness grid goes to the Online Appendix, cited in one line of main text.

Referee pushback patterns and the JOP fix

• "Your DID uses naive TWFE on staggered adoption." Re-estimate with a heterogeneity-robust estimator, show the event-study plot, and decompose the two-way estimate so the negative-weight problem is resolved.
• "Standard errors do not reflect the design." Cluster at the assignment level — the state in the AVR example — with wild-cluster bootstrap when states are few.
• "This interaction looks fished." Show the pre-registered family and the adjusted p-value; concede a null openly.

Output format

【Primary result】estimand + magnitude + uncertainty
【Robustness】each check ↔ the threat it answers (main vs appendix)
【Reproducible】master script + seeds + pinned versions + codebook? [Y/N]
【Numbers match】text == deposited output? [Y/N]
【Page discipline】main text lean, overflow in appendix? [Y/N]
【Next】jop-tables-figures

Supplementary resources

• ../../resources/external_tools.md — estimation packages and reproducibility tooling (renv, seeds, version pinning)
• ../../resources/official-source-map.md — JOP replicability-contingent acceptance and replication-analyst check