brycewang-stanford Data Analysis (psci-data-analysis)
Psychological Science holds analyses to high credibility standards: effect sizes with confidence intervals for major results, full disclosure of how the data were handled, and a clean confirmatory vs. exploratory separation. Analysis scripts and data are shared and can be checked.
When to trigger
- Running and reporting the main and supporting analyses
- A reviewer asked for effect sizes, intervals, robustness, or disclosure
- Reconciling preregistered analyses with exploratory follow-ups
- Preparing analysis scripts and a data dictionary for deposit
Reporting norms Psychological Science expects
- Effect sizes + uncertainty. Report a standardized or unstandardized effect size and a measure of uncertainty (e.g., confidence intervals) for major results — not just p-values and stars.
- Full disclosure (the "21-word-solution" spirit). Report how sample size was determined, all data exclusions (and reasons), all manipulations/conditions, and all measures. Total excluded observations must be stated.
- Confirmatory vs. exploratory. Label preregistered confirmatory analyses separately from exploratory ones; do not present exploratory results as predicted (no HARKing).
- Appropriate inference. Justify the model; report assumptions/diagnostics; correct for multiple comparisons when testing many outcomes; consider robust or Bayesian alternatives where apt.
- Replicability of the analysis. Provide analysis scripts and a data dictionary; results should
regenerate from the shared data in a fresh session (see
psci-open-science-and-transparency).
Robustness
- Show the result survives reasonable alternative specifications and exclusion choices; report sensitivity rather than a single fragile model. For small samples, be candid about uncertainty.
Worked micro-example (illustrative numbers)
A preregistered two-study package on selective attention. Study 1 (N = 240, between-subjects) tests whether a brief mindfulness induction reduces attentional capture by emotional distractors. The confirmatory analysis is a single preregistered contrast on reaction-time cost.
Confirmatory (preregistered) — Study 1
Effect: induction vs. control on capture cost (ms)
d = 0.34, 95% CI [0.08, 0.59], t(238) = 2.66, p = .008
Sensitivity: holds with/without the 6 preregistered RT-outlier exclusions
(d shifts 0.34 → 0.31), and under log-RT (d = 0.33)
Exploratory (labeled) — Study 1
Trait-anxiety × condition interaction surfaced post hoc; reported as
exploratory, flagged for confirmation in Study 2's preregistration
Confirmatory (preregistered) — Study 2 (N = 300, direct + extension)
Replicates direct effect (d = 0.29, 95% CI [0.06, 0.51]) and
preregisters the anxiety moderation that was exploratory in Study 1
Why this passes Psychological Science scrutiny: every confirmatory number carries an effect size and a CI; the anxiety interaction is honestly demoted to exploratory and then promoted to confirmatory only after preregistration in Study 2; the sensitivity line pre-empts the "fragile-to-exclusions" reviewer.
Analysis-stage reviewer pushback and the venue fix
| Reviewer pushback | What it signals here | Psychological Science fix |
|---|---|---|
| "p = .048 — too close to the line, and the CI nearly spans zero" | post-credibility-revolution distrust of just-significant single tests | report the CI prominently, add the Study 2 replication, lead with the pooled estimate |
| "Which exclusions were preregistered?" | suspicion of undisclosed forking paths | give the disclosure table: rule, count, preregistered vs. post hoc, and the estimate with vs. without |
| "Means hide the distribution" | bar-of-means aesthetic distrusted | recompute and show effect size + CI; route exhibit to psci-tables-figures |
| "Is this confirmatory?" | HARKing concern | point to the preregistration timestamp; relabel anything generated after data as exploratory |
| "Reviewer 2 could not rerun your code" | reproducibility gate | ship a fresh-session run log; see psci-open-science-and-transparency |
Calibration anchors
- One adequately powered effect with a tight CI beats three stars on an underpowered model — the journal's cautionary history is flashy-but-fragile single studies.
- Prefer estimation language ("the induction reduced capture cost by ~0.3 SD, 95% CI [...]") to dichotomous "significant/not." Bare p-value sentences read as pre-reform here.
- When N is modest, state the smallest effect the design could detect rather than implying precision you do not have; hedge magnitude claims to what the interval supports.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. Psychological Science is short-format experimental psychology with strong open-science norms; preregister, run randomization inference, and report effect sizes with family-wise corrections.
- Many outcomes / specifications:
romano_wolf(step-down FWER) orbenjamini_hochberg— report the adjusted threshold. - OVB sensitivity:
oster_delta/sensemakr. - Inference:
wild_cluster_bootstrap(few clusters),twoway_cluster/conley; multilevel data → cluster at the right level. - Re-fit off one handle:
audit_result(result_id)lists the missing checks and the exactsuggest_functionfor each. - Exhibits:
etable/did_summary_to_latexfrom the handle — no retyped numbers.
Keep the decisive checks in the body and the exhaustive battery in the supplement. See the executed chain in the JF execution walkthrough.
Anti-patterns
- p-values and stars with no effect size or confidence interval
- Selectively reporting conditions, measures, or exclusions (undisclosed flexibility)
- HARKing exploratory findings into confirmatory hypotheses
- Optional-stopping / garden-of-forking-paths analyses presented as planned
- Analysis code that does not reproduce the reported numbers
Output format
【Main result】effect size + confidence interval + meaning
【Disclosure】N-determination + all exclusions + all conditions + all measures reported? [Y/N]
【Confirmatory vs exploratory】clearly separated? [Y/N]
【Inference】assumptions/diagnostics, MHT handled?
【Reproducible】scripts + data dictionary + fresh-session check? [Y/N]
【Next】psci-tables-figures
Supplementary resources
../../resources/external_tools.md—effectsize,emmeans,metafor, JASP/jamovi, reproducible-report tooling../../resources/official-source-map.md— statistical and disclosure requirements