Data Analysis (ajps-data-analysis)
AJPS reviewers are methodologically sophisticated, and after acceptance an independent third-party
verifier re-runs your code against the numbers in the main text before the article is published (see
ajps-replication-and-verification). Analyze as if both facts are true — because they are. This skill
covers execution and reporting; design choices live in ajps-research-design.
When to trigger
- Running main and supporting analyses; building the results section
- A reviewer asked for robustness, heterogeneity, or alternative specifications
- Reconciling preregistered vs. exploratory analyses
- Making the analysis reproducible so the verifier's re-run will match
Analysis norms AJPS expects
- Report uncertainty and magnitude. Confidence/credible intervals and substantive effect sizes,
not just significance stars — say what the estimate means.
- Robustness that probes, not decorates. Show specifications that could break the result
(alternative measures, samples, estimators, fixed effects) and say what you learn.
- Heterogeneity with discipline. Pre-specify subgroups where possible; adjust for multiple
comparisons; do not mine for a significant interaction and theorize it post hoc.
- Right inference. Cluster at the assignment/sampling level; randomization inference for
experiments; wild-cluster bootstrap when clusters are few.
- Preregistration discipline. Separate registered from exploratory analyses; justify any
deviation from the plan.
- Measurement. Validate constructs; report reliability; show results are not an artifact of a
single coding/scaling choice.
Computational / text-as-data specifics
- Document model/version, hyperparameters, seeds, and validation against human-labeled samples.
- For topic models/embeddings/LLM pipelines: report stability and a validation step; do not treat raw
outputs as ground truth.
Reproducibility while you work (so the verifier's re-run matches)
- One master script regenerates every table and figure from the (raw or constructed) data, in
order, setting the working directory once.
- Set and report seeds for every stochastic step (bootstrap, randomization inference, simulation,
jittered plots) — the verifier needs identical draws.
- Record exact software versions (e.g., "R 4.3.2", "Stata/MP 18.0") and pin packages
(
renv.lock / requirements.txt / logged ssc/net installs).
- Keep the manuscript's table/figure numbers matched to script outputs — the verifier checks the
numerical results in the main text line by line.
Analysis-decision checklist a quantitative AJPS referee runs
| Question the referee asks |
Pass condition |
Fix if it fails |
| Does the estimator recover the stated estimand? |
Estimand named; estimator matches |
Name the target quantity before the table |
| Is inference at the right level? |
Clustered at assignment/sampling level |
Re-cluster; wild-cluster bootstrap if few clusters |
| Will the verifier's re-run match the printed numbers? |
Master script regenerates every exhibit |
Script everything; set seeds; pin versions |
Worked micro-example (illustrative numbers)
A survey experiment tests whether a co-partisan endorsement raises policy support. The pre-analysis plan
names the estimand (ITT on a 0-100 scale), the primary contrast, and one moderator (political knowledge).
Result: +7.4 points (95% CI 3.1-11.7), randomization-inference p = 0.004 (illustrative). A knowledge
interaction that was not pre-specified as confirmatory goes to an exploratory subsection, flagged, with a
multiple-comparison note. Every number is emitted by one seeded master script, so the AJPS verifier's
re-run reproduces the main-text figures exactly.
Referee-pushback patterns and the venue-specific fix
- "Identification leans on selection-on-observables." -> Report an unobserved-confounder sensitivity
bound (how strong a confounder must be to overturn the estimate); soften causal language if fragile.
- "This interaction looks mined." -> Show it was pre-registered, or relabel it exploratory and adjust for
multiple comparisons; never HARK it into a hypothesis.
- "I cannot reproduce Table 2 from your code." -> Fatal at AJPS verification; rebuild the master script so
every number regenerates with fixed seeds and pinned versions.
Calibration anchor: AJPS's independent verifier re-runs deposited code against the main-text numbers before
publication, so "it works on my machine" is not enough — confirm the live verification wording against the
journal's current guidelines.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. AJPS prizes credible identification across American / comparative / IR subfields; DiD/IV/RDD for observational claims, randomization inference for experiments.
- Many outcomes / specifications:
romano_wolf (step-down FWER) or
benjamini_hochberg — report the adjusted threshold.
- OVB sensitivity:
oster_delta / sensemakr.
- Inference:
wild_cluster_bootstrap (few clusters), twoway_cluster / conley;
multilevel data → cluster at the right level.
- Re-fit off one handle:
audit_result(result_id) lists the missing checks and the
exact suggest_function for each.
- Exhibits:
etable / did_summary_to_latex from the handle — no retyped numbers.
Keep the decisive checks in the body and the exhaustive battery in the supplement. See
the executed chain in the JF execution walkthrough.
Anti-patterns
- Stars-only tables with no effect sizes or intervals
- "Robustness" that only reruns near-identical specs to manufacture stability
- p-hacking / fishing for a significant interaction; HARKing exploratory results into hypotheses
- Clustering at the wrong level or ignoring few-cluster problems
- Hand-edited numbers in the manuscript that the deposited code cannot regenerate (verification fails)
Output format
【Main estimate】magnitude + interval + substantive meaning
【Identification check】(per research-design) result
【Robustness】specs that could break it -> what held
【Heterogeneity】pre-specified? MHT-adjusted?
【Registered vs exploratory】clearly separated?
【Reproducible】master script + seeds + pinned versions, numbers match? [Y/N]
【Next】ajps-tables-figures
Supplementary resources
1---2name: ajps-data-analysis3description: Use when executing and reporting the analysis for an American Journal of Political Science (AJPS) manuscript. AJPS will have a third-party verifier re-run your exact code against the numerical results in the main text before publication, so analyze reproducibly from the first line. Guides analysis and reporting norms; it does not fabricate results.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/ajps-data-analysis6---78# Data Analysis (ajps-data-analysis)910AJPS reviewers are methodologically sophisticated, and after acceptance an **independent third-party11verifier re-runs your code** against the numbers in the main text before the article is published (see12`ajps-replication-and-verification`). Analyze as if both facts are true — because they are. This skill13covers execution and reporting; design choices live in `ajps-research-design`.1415## When to trigger1617- Running main and supporting analyses; building the results section18- A reviewer asked for robustness, heterogeneity, or alternative specifications19- Reconciling preregistered vs. exploratory analyses20- Making the analysis reproducible so the verifier's re-run will match2122## Analysis norms AJPS expects23241. **Report uncertainty and magnitude.** Confidence/credible intervals and substantive effect sizes,25 not just significance stars — say what the estimate *means*.262. **Robustness that probes, not decorates.** Show specifications that could *break* the result27 (alternative measures, samples, estimators, fixed effects) and say what you learn.283. **Heterogeneity with discipline.** Pre-specify subgroups where possible; adjust for multiple29 comparisons; do not mine for a significant interaction and theorize it post hoc.304. **Right inference.** Cluster at the assignment/sampling level; randomization inference for31 experiments; wild-cluster bootstrap when clusters are few.325. **Preregistration discipline.** Separate **registered** from **exploratory** analyses; justify any33 deviation from the plan.346. **Measurement.** Validate constructs; report reliability; show results are not an artifact of a35 single coding/scaling choice.3637## Computational / text-as-data specifics38- Document model/version, hyperparameters, seeds, and validation against human-labeled samples.39- For topic models/embeddings/LLM pipelines: report stability and a validation step; do not treat raw40 outputs as ground truth.4142## Reproducibility while you work (so the verifier's re-run matches)43- One **master script** regenerates every table and figure from the (raw or constructed) data, in44 order, setting the working directory once.45- **Set and report seeds** for every stochastic step (bootstrap, randomization inference, simulation,46 jittered plots) — the verifier needs identical draws.47- Record exact **software versions** (e.g., "R 4.3.2", "Stata/MP 18.0") and pin packages48 (`renv.lock` / `requirements.txt` / logged `ssc`/`net` installs).49- Keep the manuscript's table/figure numbers matched to script outputs — the verifier checks the50 **numerical results in the main text** line by line.5152## Analysis-decision checklist a quantitative AJPS referee runs5354| Question the referee asks | Pass condition | Fix if it fails |55|---------------------------|----------------|-----------------|56| Does the estimator recover the stated estimand? | Estimand named; estimator matches | Name the target quantity before the table |57| Is inference at the right level? | Clustered at assignment/sampling level | Re-cluster; wild-cluster bootstrap if few clusters |58| Will the verifier's re-run match the printed numbers? | Master script regenerates every exhibit | Script everything; set seeds; pin versions |5960## Worked micro-example (illustrative numbers)6162A survey experiment tests whether a co-partisan endorsement raises policy support. The pre-analysis plan63names the estimand (ITT on a 0-100 scale), the primary contrast, and one moderator (political knowledge).64Result: **+7.4 points (95% CI 3.1-11.7), randomization-inference p = 0.004** *(illustrative)*. A knowledge65interaction that was *not* pre-specified as confirmatory goes to an exploratory subsection, flagged, with a66multiple-comparison note. Every number is emitted by one seeded master script, so the AJPS verifier's67re-run reproduces the main-text figures exactly.6869## Referee-pushback patterns and the venue-specific fix7071- *"Identification leans on selection-on-observables."* -> Report an unobserved-confounder sensitivity72 bound (how strong a confounder must be to overturn the estimate); soften causal language if fragile.73- *"This interaction looks mined."* -> Show it was pre-registered, or relabel it exploratory and adjust for74 multiple comparisons; never HARK it into a hypothesis.75- *"I cannot reproduce Table 2 from your code."* -> Fatal at AJPS verification; rebuild the master script so76 every number regenerates with fixed seeds and pinned versions.7778Calibration anchor: AJPS's independent verifier re-runs deposited code against the main-text numbers before79publication, so "it works on my machine" is not enough — confirm the live verification wording against the80journal's current guidelines.8182## Execution bridge (StatsPAI / Stata MCP)8384Run the battery, don't just enumerate it. Full map:85[`execution-with-mcp`](../../../shared-resources/empirical-methods/execution-with-mcp.md). AJPS prizes credible identification across American / comparative / IR subfields; DiD/IV/RDD for observational claims, randomization inference for experiments.8687- **Many outcomes / specifications:** `romano_wolf` (step-down FWER) or88 `benjamini_hochberg` — report the adjusted threshold.89- **OVB sensitivity:** `oster_delta` / `sensemakr`.90- **Inference:** `wild_cluster_bootstrap` (few clusters), `twoway_cluster` / `conley`;91 multilevel data → cluster at the right level.92- **Re-fit off one handle:** `audit_result(result_id)` lists the missing checks and the93 exact `suggest_function` for each.94- **Exhibits:** `etable` / `did_summary_to_latex` from the handle — no retyped numbers.9596Keep the decisive checks in the body and the exhaustive battery in the supplement. See97the executed chain in the [JF execution walkthrough](../../../Journal-of-Finance-Skills/resources/worked-examples/02-execution-walkthrough.md).98## Anti-patterns99100- Stars-only tables with no effect sizes or intervals101- "Robustness" that only reruns near-identical specs to manufacture stability102- p-hacking / fishing for a significant interaction; HARKing exploratory results into hypotheses103- Clustering at the wrong level or ignoring few-cluster problems104- Hand-edited numbers in the manuscript that the deposited code cannot regenerate (verification fails)105106## Output format107108```109【Main estimate】magnitude + interval + substantive meaning110【Identification check】(per research-design) result111【Robustness】specs that could break it -> what held112【Heterogeneity】pre-specified? MHT-adjusted?113【Registered vs exploratory】clearly separated?114【Reproducible】master script + seeds + pinned versions, numbers match? [Y/N]115【Next】ajps-tables-figures116```117118## Supplementary resources119120- [`../../resources/external_tools.md`](../../resources/external_tools.md) — estimation, inference, and text-as-data packages121- [`../../resources/official-source-map.md`](../../resources/official-source-map.md) — third-party verification of numerical results