Data Analysis (poq-data-analysis)
POQ reviewers are methodologically sophisticated, and the journal requires replication materials that
reproduce exactly all published tables and figures (see poq-transparency-and-data-policy).
Analyze as if both are true — because they are. The defining POQ demand is design-based inference:
survey weights, strata, and clusters belong in the variance estimator, not just the point estimate.
Design decisions live in poq-survey-design-and-measurement.
When to trigger
- Running main and supporting analyses; building the results section
- A reviewer asked for design-based SEs, robustness, or alternative weighting
- Reconciling preregistered vs. exploratory analyses
- Making the analysis reproducible before deposit
Analysis norms POQ expects
- Design-based inference. Use complex-survey estimators (
svy: / survey / samplics); declare
weights, strata, and PSUs. Report the design effect (DEFF); do not present naive IID standard
errors on a clustered, weighted sample.
- Report uncertainty honestly. Confidence intervals, not just stars; the magnitude and
substantive meaning of the estimate. For opinion shares, show the margin of error and the
definition of precision.
- Weighted vs. unweighted. Show both where they diverge and explain why; do not let weighting
silently drive the headline result.
- Robustness that probes, not decorates. Alternative weighting/calibration, alternative codings,
sensitivity to nonresponse assumptions, mode controls — specs that could break the result.
- Heterogeneity with discipline. Pre-specify subgroups; correct for multiple comparisons; do not
mine for a significant interaction and theorize it post hoc.
- Measurement carries through. Show the result is not an artifact of a coding/scaling choice;
report reliability and, where relevant, measurement invariance across groups/modes.
Missing data, nonresponse & trends
- Distinguish item nonresponse handling (multiple imputation vs. listwise) and report the choice.
- Adjust for nonresponse explicitly; state assumptions (MAR vs. not) and probe sensitivity.
- For trend/Polls-in-Context analyses, hold question wording and mode constant or flag the break.
Reproducibility while you work (not at the end)
- One master script regenerates every table and figure from the (raw or constructed) data.
- Set and report seeds for bootstrap, multiple imputation, simulation, and any stochastic step.
- Pin software/package versions (
renv.lock, requirements.txt, recorded ssc/net installs).
- Keep table/figure numbers matched to script outputs — POQ re-runs the package against the exhibits.
POQ replication acceptance gate
Before writing results prose, run a local replication gate that mirrors the journal's expectations:
| Gate |
Required evidence |
| Survey design object |
A single declared object listing weights, strata, PSUs, finite-population corrections where used, and missing-data handling. |
| Table/figure manifest |
Every exhibit has a script target, input data file, and output path; no manual spreadsheet edits. |
| Sensitivity queue |
Alternative weights, nonresponse adjustments, mode/question wording breaks, and subgroup corrections listed before results are interpreted. |
| Exact reproduction |
A clean checkout or temporary directory regenerates all published numbers with matching rounding. |
Only after this gate should the manuscript claim that the analysis is reproducible. POQ readers notice
when design-based inference is described in text but not actually encoded in the scripts.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. Public Opinion Quarterly is survey methodology and public opinion; the chain serves causal/experimental claims, while survey-design and measurement contributions use their own standards (sampling, weighting, measurement error).
- Many outcomes / specifications:
romano_wolf (step-down FWER) or
benjamini_hochberg — report the adjusted threshold.
- OVB sensitivity:
oster_delta / sensemakr.
- Inference:
wild_cluster_bootstrap (few clusters), twoway_cluster / conley;
multilevel data → cluster at the right level.
- Re-fit off one handle:
audit_result(result_id) lists the missing checks and the
exact suggest_function for each.
- Exhibits:
etable / did_summary_to_latex from the handle — no retyped numbers.
Keep the decisive checks in the body and the exhaustive battery in the supplement. See
the executed chain in the JF execution walkthrough.
Anti-patterns
- Naive IID standard errors on a weighted, clustered survey (the most common POQ analysis flaw)
- Stars-only tables with no effect sizes, intervals, or margins of error
- Weighting the estimate but ignoring the weights/design in the variance
- p-hacking / HARKing exploratory subgroup results into hypotheses
- A results section whose numbers the deposited code cannot reproduce
Output format
【Estimator】complex-survey (weights/strata/PSUs declared)? [Y/N]
【Main estimate】magnitude + interval/MOE + substantive meaning
【Design effect】DEFF reported?
【Weighted vs unweighted】reconciled where they differ?
【Robustness】specs that could break it → what held
【Heterogeneity】pre-specified? MHT-adjusted?
【Reproducible】master script + seeds + pinned versions? [Y/N]
【Next】poq-tables-figures
Supplementary resources
Source: brycewang-stanford/Awesome-Journal-Skills → Public-Opinion-Quarterly-Skills/skills/poq-data-analysis/SKILL.md
1---2name: poq-data-analysis3description: Use when executing and reporting the analysis for a Public Opinion Quarterly (POQ) manuscript so it survives expert, double-blind review — design-based inference that respects survey weights, strata, and clusters, honest uncertainty, robustness, and reproducibility. POQ verifies that code reproduces every table and figure. Guides analysis norms; it does not fabricate results.4---5
6
7# Data Analysis (poq-data-analysis)
8
9POQ reviewers are methodologically sophisticated, and the journal requires replication materials that
10**reproduce exactly all published tables and figures** (see `poq-transparency-and-data-policy`).
11Analyze as if both are true — because they are. The defining POQ demand is **design-based inference**:
12survey weights, strata, and clusters belong in the variance estimator, not just the point estimate.
13Design decisions live in `poq-survey-design-and-measurement`.
14
15## When to trigger
16
17- Running main and supporting analyses; building the results section
18- A reviewer asked for design-based SEs, robustness, or alternative weighting
19- Reconciling preregistered vs. exploratory analyses
20- Making the analysis reproducible before deposit
21
22## Analysis norms POQ expects
23
241. **Design-based inference.** Use complex-survey estimators (`svy:` / `survey` / `samplics`); declare
25 weights, strata, and PSUs. Report the **design effect (DEFF)**; do not present naive IID standard
26 errors on a clustered, weighted sample.
272. **Report uncertainty honestly.** Confidence intervals, not just stars; the magnitude and
28 substantive meaning of the estimate. For opinion shares, show the margin of error and the
29 definition of precision.
303. **Weighted vs. unweighted.** Show both where they diverge and explain why; do not let weighting
31 silently drive the headline result.
324. **Robustness that probes, not decorates.** Alternative weighting/calibration, alternative codings,
33 sensitivity to nonresponse assumptions, mode controls — specs that could *break* the result.
345. **Heterogeneity with discipline.** Pre-specify subgroups; correct for multiple comparisons; do not
35 mine for a significant interaction and theorize it post hoc.
366. **Measurement carries through.** Show the result is not an artifact of a coding/scaling choice;
37 report reliability and, where relevant, measurement invariance across groups/modes.
38
39## Missing data, nonresponse & trends
40- Distinguish item nonresponse handling (multiple imputation vs. listwise) and report the choice.
41- Adjust for nonresponse explicitly; state assumptions (MAR vs. not) and probe sensitivity.
42- For trend/Polls-in-Context analyses, hold question wording and mode constant or flag the break.
43
44## Reproducibility while you work (not at the end)
45- One **master script** regenerates every table and figure from the (raw or constructed) data.
46- **Set and report seeds** for bootstrap, multiple imputation, simulation, and any stochastic step.
47- Pin software/package versions (`renv.lock`, `requirements.txt`, recorded `ssc`/`net` installs).
48- Keep table/figure numbers matched to script outputs — POQ re-runs the package against the exhibits.
49
50## POQ replication acceptance gate
51
52Before writing results prose, run a local replication gate that mirrors the journal's expectations:
53
54| Gate | Required evidence |
55|------|-------------------|
56| Survey design object | A single declared object listing weights, strata, PSUs, finite-population corrections where used, and missing-data handling. |
57| Table/figure manifest | Every exhibit has a script target, input data file, and output path; no manual spreadsheet edits. |
58| Sensitivity queue | Alternative weights, nonresponse adjustments, mode/question wording breaks, and subgroup corrections listed before results are interpreted. |
59| Exact reproduction | A clean checkout or temporary directory regenerates all published numbers with matching rounding. |
60
61Only after this gate should the manuscript claim that the analysis is reproducible. POQ readers notice
62when design-based inference is described in text but not actually encoded in the scripts.
63
64## Execution bridge (StatsPAI / Stata MCP)
65
66Run the battery, don't just enumerate it. Full map:
67[`execution-with-mcp`](../../../shared-resources/empirical-methods/execution-with-mcp.md). Public Opinion Quarterly is survey methodology and public opinion; the chain serves causal/experimental claims, while survey-design and measurement contributions use their own standards (sampling, weighting, measurement error).
68
69- **Many outcomes / specifications:** `romano_wolf` (step-down FWER) or
70 `benjamini_hochberg` — report the adjusted threshold.
71- **OVB sensitivity:** `oster_delta` / `sensemakr`.
72- **Inference:** `wild_cluster_bootstrap` (few clusters), `twoway_cluster` / `conley`;
73 multilevel data → cluster at the right level.
74- **Re-fit off one handle:** `audit_result(result_id)` lists the missing checks and the
75 exact `suggest_function` for each.
76- **Exhibits:** `etable` / `did_summary_to_latex` from the handle — no retyped numbers.
77
78Keep the decisive checks in the body and the exhaustive battery in the supplement. See
79the executed chain in the [JF execution walkthrough](../../../Journal-of-Finance-Skills/resources/worked-examples/02-execution-walkthrough.md).
80## Anti-patterns
81
82- Naive IID standard errors on a weighted, clustered survey (the most common POQ analysis flaw)
83- Stars-only tables with no effect sizes, intervals, or margins of error
84- Weighting the estimate but ignoring the weights/design in the variance
85- p-hacking / HARKing exploratory subgroup results into hypotheses
86- A results section whose numbers the deposited code cannot reproduce
87
88## Output format
89
90```
91【Estimator】complex-survey (weights/strata/PSUs declared)? [Y/N]
92【Main estimate】magnitude + interval/MOE + substantive meaning
93【Design effect】DEFF reported?
94【Weighted vs unweighted】reconciled where they differ?
95【Robustness】specs that could break it → what held
96【Heterogeneity】pre-specified? MHT-adjusted?
97【Reproducible】master script + seeds + pinned versions? [Y/N]
98【Next】poq-tables-figures
99```
100
101## Supplementary resources
102
103- [`../../resources/external_tools.md`](../../resources/external_tools.md) — complex-survey estimation, weighting, imputation packages
104- [`../../resources/official-source-map.md`](../../resources/official-source-map.md) — reproducibility / replication-archive policy
105
106---
107
108**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `Public-Opinion-Quarterly-Skills/skills/poq-data-analysis/SKILL.md`
thedixitjain