Data Analysis (jop-data-analysis)
At JOP, analysis and reproducibility are the same task: acceptance is contingent on
replicability, and a JOP replication analyst re-runs your code at conditional acceptance. Write the
analysis so that every number in the paper is regenerated by a script — and reported with honest
uncertainty within the page budget.
When to trigger
- Setting up the estimation/analysis pipeline
- Deciding which robustness checks belong in the main text vs the Online Appendix
- A reviewer asked for additional specifications, uncertainty, or sensitivity
- Preparing numbers that must match the deposited replication package exactly
Analysis norms
- Report uncertainty, not just point estimates: CIs, SEs (clustered appropriately), and
substantive effect sizes a general reader can interpret.
- Specification transparency: show the primary specification clearly; relegate the grid of
alternatives to the Online Appendix, but reference it.
- Robustness that targets the threat: each check should answer a specific objection (confounding,
functional form, sample, measurement), not pad the count.
- Multiple comparisons: adjust or pre-specify when testing many implications.
- Substantive interpretation: translate coefficients into quantities of interest (predicted
probabilities, marginal effects) — general-interest readers want magnitudes, not just stars.
Reproducible-from-line-one (the JOP analyst will re-run this)
- One master script runs everything in order and sets the working directory once.
- Set a seed for every stochastic step (bootstrap, simulation, MCMC, jitter, sampling).
- Record software and package versions for the readme (e.g., "R 4.3.1", "Stata/MP 18.0").
- Build a codebook naming and defining every variable used in the analysis.
- Tables and figures are generated by code, never hand-edited — numbers in the text must match.
Fit the analysis to the page budget
- Lead with the result that carries the argument; do not narrate every regression.
- Move the robustness grid, balance tables, and diagnostics to the Online Appendix (≤ 25 pp).
- A Short Article (≤ 10 pp) should show one clean, decisive analysis, not a buffet.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. Journal of Politics spans observational and experimental political science; report the identifying assumption and the magnitude, not just stars.
- Many outcomes / specifications:
romano_wolf (step-down FWER) or
benjamini_hochberg — report the adjusted threshold.
- OVB sensitivity:
oster_delta / sensemakr.
- Inference:
wild_cluster_bootstrap (few clusters), twoway_cluster / conley;
multilevel data → cluster at the right level.
- Re-fit off one handle:
audit_result(result_id) lists the missing checks and the
exact suggest_function for each.
- Exhibits:
etable / did_summary_to_latex from the handle — no retyped numbers.
Keep the decisive checks in the body and the exhaustive battery in the supplement. See
the executed chain in the JF execution walkthrough.
Anti-patterns
- Numbers in the manuscript that the deposited code cannot reproduce (fails the analyst check)
- Unseeded randomness or unpinned versions ("works on my machine")
- Star-gazing with no effect sizes or uncertainty a general reader can use
- Robustness checks chosen to inflate the count rather than rebut a threat
- Cramming every specification into the main text and blowing the page budget
What a JOP analysis referee is looking for
The reviewer pool spans subfields, so an analysis only a specialist can audit reads as fragile. Map each
demand to the move that satisfies it before the page count forces an ugly cut.
| Referee demand |
Pass move |
Fail signal |
| Usable magnitude |
Marginal effect or predicted probability with CI |
Coefficient stars, no magnitude in prose |
| Correct uncertainty |
Cluster at assignment level; randomization inference |
Default SEs on clustered or experimental data |
| Targeted robustness |
Each check named to the threat it rebuts |
A grid with no mapping to objections |
| Multiplicity honesty |
Pre-specified families; adjusted p-values |
One mined "significant" interaction |
| Reproducibility |
Master script regenerates every number |
"Available on request"; drifting numbers |
Worked micro-example (illustrative figures)
A hypothetical Short Article asks whether a state's adoption of automatic voter registration (AVR) raised
turnout, using a staggered difference-in-differences across states. The first pass runs naive two-way
fixed effects and reports a +3.1-point effect (illustrative). Because adoption is staggered, already-treated
states act as forbidden controls and the estimate carries negative-weight comparisons. The JOP-credible
re-analysis uses a heterogeneity-robust estimator (Callaway–Sant'Anna or Sun–Abraham), reports the
group-time average as +1.8 points, 95% CI [0.4, 3.2] (illustrative), shows flat pre-trends, and clusters
by state. The robustness grid goes to the Online Appendix, cited in one line of main text.
Referee pushback patterns and the JOP fix
- "Your DID uses naive TWFE on staggered adoption." Re-estimate with a heterogeneity-robust estimator,
show the event-study plot, and decompose the two-way estimate so the negative-weight problem is resolved.
- "Standard errors do not reflect the design." Cluster at the assignment level — the state in the AVR
example — with wild-cluster bootstrap when states are few.
- "This interaction looks fished." Show the pre-registered family and the adjusted p-value; concede a
null openly.
Output format
【Primary result】estimand + magnitude + uncertainty
【Robustness】each check ↔ the threat it answers (main vs appendix)
【Reproducible】master script + seeds + pinned versions + codebook? [Y/N]
【Numbers match】text == deposited output? [Y/N]
【Page discipline】main text lean, overflow in appendix? [Y/N]
【Next】jop-tables-figures
Supplementary resources
Source: brycewang-stanford/Awesome-Journal-Skills → Journal-of-Politics-Skills/skills/jop-data-analysis/SKILL.md
1---2name: jop-data-analysis3description: Use for analysis-stage decisions on a The Journal of Politics (JOP) manuscript — uncertainty, robustness, and reporting norms — written so the work is reproducible from line one. JOP makes acceptance contingent on replicability and a JOP replication analyst re-runs the code, so every reported number must come from a script. Guides analysis; it does not fabricate results.4---5
6
7# Data Analysis (jop-data-analysis)
8
9At JOP, analysis and **reproducibility are the same task**: acceptance is **contingent on
10replicability**, and a **JOP replication analyst** re-runs your code at conditional acceptance. Write the
11analysis so that every number in the paper is **regenerated by a script** — and reported with honest
12uncertainty within the **page budget**.
13
14## When to trigger
15
16- Setting up the estimation/analysis pipeline
17- Deciding which robustness checks belong in the main text vs the Online Appendix
18- A reviewer asked for additional specifications, uncertainty, or sensitivity
19- Preparing numbers that must match the deposited replication package exactly
20
21## Analysis norms
22
23- **Report uncertainty**, not just point estimates: CIs, SEs (clustered appropriately), and
24 substantive effect sizes a general reader can interpret.
25- **Specification transparency**: show the primary specification clearly; relegate the grid of
26 alternatives to the Online Appendix, but reference it.
27- **Robustness that targets the threat**: each check should answer a specific objection (confounding,
28 functional form, sample, measurement), not pad the count.
29- **Multiple comparisons**: adjust or pre-specify when testing many implications.
30- **Substantive interpretation**: translate coefficients into quantities of interest (predicted
31 probabilities, marginal effects) — general-interest readers want magnitudes, not just stars.
32
33## Reproducible-from-line-one (the JOP analyst will re-run this)
34
35- One **master script** runs everything in order and sets the working directory once.
36- **Set a seed** for every stochastic step (bootstrap, simulation, MCMC, jitter, sampling).
37- **Record software and package versions** for the readme (e.g., "R 4.3.1", "Stata/MP 18.0").
38- Build a **codebook** naming and defining every variable used in the analysis.
39- Tables and figures are **generated by code**, never hand-edited — numbers in the text must match.
40
41## Fit the analysis to the page budget
42
43- Lead with the result that carries the argument; do not narrate every regression.
44- Move the robustness grid, balance tables, and diagnostics to the **Online Appendix (≤ 25 pp)**.
45- A Short Article (≤ 10 pp) should show one clean, decisive analysis, not a buffet.
46
47## Execution bridge (StatsPAI / Stata MCP)
48
49Run the battery, don't just enumerate it. Full map:
50[`execution-with-mcp`](../../../shared-resources/empirical-methods/execution-with-mcp.md). Journal of Politics spans observational and experimental political science; report the identifying assumption and the magnitude, not just stars.
51
52- **Many outcomes / specifications:** `romano_wolf` (step-down FWER) or
53 `benjamini_hochberg` — report the adjusted threshold.
54- **OVB sensitivity:** `oster_delta` / `sensemakr`.
55- **Inference:** `wild_cluster_bootstrap` (few clusters), `twoway_cluster` / `conley`;
56 multilevel data → cluster at the right level.
57- **Re-fit off one handle:** `audit_result(result_id)` lists the missing checks and the
58 exact `suggest_function` for each.
59- **Exhibits:** `etable` / `did_summary_to_latex` from the handle — no retyped numbers.
60
61Keep the decisive checks in the body and the exhaustive battery in the supplement. See
62the executed chain in the [JF execution walkthrough](../../../Journal-of-Finance-Skills/resources/worked-examples/02-execution-walkthrough.md).
63## Anti-patterns
64
65- Numbers in the manuscript that the deposited code cannot reproduce (fails the analyst check)
66- Unseeded randomness or unpinned versions ("works on my machine")
67- Star-gazing with no effect sizes or uncertainty a general reader can use
68- Robustness checks chosen to inflate the count rather than rebut a threat
69- Cramming every specification into the main text and blowing the page budget
70
71## What a JOP analysis referee is looking for
72
73The reviewer pool spans subfields, so an analysis only a specialist can audit reads as fragile. Map each
74demand to the move that satisfies it before the page count forces an ugly cut.
75
76| Referee demand | Pass move | Fail signal |
77|----------------|-----------|-------------|
78| Usable magnitude | Marginal effect or predicted probability with CI | Coefficient stars, no magnitude in prose |
79| Correct uncertainty | Cluster at assignment level; randomization inference | Default SEs on clustered or experimental data |
80| Targeted robustness | Each check named to the threat it rebuts | A grid with no mapping to objections |
81| Multiplicity honesty | Pre-specified families; adjusted p-values | One mined "significant" interaction |
82| Reproducibility | Master script regenerates every number | "Available on request"; drifting numbers |
83
84## Worked micro-example (illustrative figures)
85
86A hypothetical Short Article asks whether a state's adoption of automatic voter registration (AVR) raised
87turnout, using a staggered difference-in-differences across states. The first pass runs naive two-way
88fixed effects and reports a +3.1-point effect (illustrative). Because adoption is staggered, already-treated
89states act as forbidden controls and the estimate carries negative-weight comparisons. The JOP-credible
90re-analysis uses a heterogeneity-robust estimator (Callaway–Sant'Anna or Sun–Abraham), reports the
91group-time average as +1.8 points, 95% CI [0.4, 3.2] (illustrative), shows flat pre-trends, and clusters
92by state. The robustness grid goes to the Online Appendix, cited in one line of main text.
93
94## Referee pushback patterns and the JOP fix
95
96- *"Your DID uses naive TWFE on staggered adoption."* Re-estimate with a heterogeneity-robust estimator,
97 show the event-study plot, and decompose the two-way estimate so the negative-weight problem is resolved.
98- *"Standard errors do not reflect the design."* Cluster at the assignment level — the state in the AVR
99 example — with wild-cluster bootstrap when states are few.
100- *"This interaction looks fished."* Show the pre-registered family and the adjusted p-value; concede a
101 null openly.
102
103## Output format
104
105```
106【Primary result】estimand + magnitude + uncertainty
107【Robustness】each check ↔ the threat it answers (main vs appendix)
108【Reproducible】master script + seeds + pinned versions + codebook? [Y/N]
109【Numbers match】text == deposited output? [Y/N]
110【Page discipline】main text lean, overflow in appendix? [Y/N]
111【Next】jop-tables-figures
112```
113
114## Supplementary resources
115
116- [`../../resources/external_tools.md`](../../resources/external_tools.md) — estimation packages and reproducibility tooling (renv, seeds, version pinning)
117- [`../../resources/official-source-map.md`](../../resources/official-source-map.md) — JOP replicability-contingent acceptance and replication-analyst check
118
119---
120
121**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `Journal-of-Politics-Skills/skills/jop-data-analysis/SKILL.md`
thedixitjain