Data Analysis (jedpsych-data-analysis)
The Journal of Educational Psychology holds analyses to the standards of a rigorous psychological research
journal operating in nested educational settings. The recurring requirements are: model the nesting
(students in classes in schools), report effect sizes with confidence intervals that are
educationally interpretable, test the mechanism (mediation/moderation), and disclose fully under
JARS. Analysis scripts and data are expected to be shareable and reproducible.
When to trigger
- Running and reporting the main and supporting analyses
- A reviewer asked for multilevel modeling, effect sizes, mechanism tests, or disclosure
- Reconciling preregistered analyses with exploratory follow-ups
- Preparing analysis scripts and a codebook for deposit
Reporting norms JEP expects
- Respect the nesting. Use multilevel (hierarchical linear) models, SEM, or growth models that
account for students nested in classrooms/schools. Cluster-robust or random-effects inference is
expected; ignoring clustering deflates standard errors and is a standard JEP rejection reason.
- Educationally meaningful effect sizes + uncertainty. Report a standardized effect (e.g., Hedges's
g, a multilevel d, R²/variance explained, or a growth-rate difference) with a confidence
interval, and interpret it in learning terms (e.g., months of progress, percentile shift) — not just
p-values and stars.
- Test the mechanism. JEP is theory-driven: where the hypothesis includes a learning/motivational
process, fit the mediation (with appropriate multilevel mediation methods) or moderation, not only the
total effect.
- Full disclosure (JARS). Report how sample size was determined, all conditions and measures, all
exclusions/attrition (with reasons and counts), missing-data handling (e.g., FIML/multiple imputation),
and model specification. Confirmatory vs. exploratory must be clearly separated.
- Appropriate inference. Justify the model; report assumptions/diagnostics and fit indices for SEM;
correct for multiple comparisons across many outcomes; consider robustness to alternative specifications.
Robustness and missing data
- Show the result survives reasonable alternative specifications (covariate sets, model form, with/without
exclusions). Handle attrition and missingness with principled methods (FIML, MI) and report rates by arm.
Worked micro-example (illustrative numbers)
A preregistered cluster-randomized reading-comprehension trial (48 classrooms, ~1,100 students). The
confirmatory analysis is a two-level model with a pretest covariate and a preregistered mediation test.
Confirmatory (preregistered) — primary effect
Two-level model (students within classrooms), pretest-adjusted:
classroom-level treatment effect on transfer comprehension
g = 0.23, 95% CI [0.06, 0.40]; ICC = 0.14; ~2.0 months of progress.
Inference uses random classroom intercepts; SEs respect clustering.
Confirmatory (preregistered) — mechanism
Multilevel mediation: monitoring gain mediates ~40% of the effect,
indirect 95% CI [0.02, 0.13] (excludes 0).
Sensitivity: holds with/without the preregistered attrition exclusions
(g 0.23 → 0.21), and under FIML for missing posttests.
Exploratory (labeled): larger effect for initially low-comprehension
readers (ATI); reported as exploratory, flagged for future confirmation.
Why this passes JEP scrutiny: the model respects nesting; the effect carries a CI and an educational
interpretation; the mechanism is tested, not asserted; the sensitivity line pre-empts the "fragile-to-
exclusions" reviewer; and the ATI is honestly demoted to exploratory.
Analysis-stage reviewer pushback and the venue fix
| Reviewer pushback |
What it signals here |
JEP fix |
| "You ignored clustering" |
deflated SEs from nesting |
refit a multilevel/random-effects model; report the ICC |
| "Effect size, and what does it mean for learning?" |
post-reform interpretability bar |
add a CI and an educational metric (months/percentile) |
| "Mechanism untested" |
total effect without theory |
fit the preregistered multilevel mediation/moderation |
| "Which analyses were preregistered?" |
forking-paths suspicion |
give the disclosure table; relabel post hoc as exploratory |
| "How was attrition handled?" |
missing-data validity |
report rates by arm; use FIML/MI; show robustness |
Calibration anchors
- One well-powered, properly nested effect with a tight CI and a clear educational interpretation beats a
pile of stars from a model that treated students as independent — the latter is a routine JEP reject.
- Prefer estimation language ("the intervention raised transfer comprehension by g = 0.23, ~2 months of
progress, 95% CI [...]") to dichotomous "significant/not."
- Mechanism evidence is what makes the paper educational psychology rather than evaluation; budget the
mediation/moderation test as a first-class result, not an afterthought.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. JEdPsych mixes field/lab experiments and observational school data; multilevel (student-in-class-in-school) inference and many-outcome corrections matter most.
- Many outcomes / specifications:
romano_wolf (step-down FWER) or
benjamini_hochberg — report the adjusted threshold.
- OVB sensitivity:
oster_delta / sensemakr.
- Inference:
wild_cluster_bootstrap (few clusters), twoway_cluster / conley;
multilevel data → cluster at the right level.
- Re-fit off one handle:
audit_result(result_id) lists the missing checks and the
exact suggest_function for each.
- Exhibits:
etable / did_summary_to_latex from the handle — no retyped numbers.
Keep the decisive checks in the body and the exhaustive battery in the supplement. See
the executed chain in the JF execution walkthrough.
Anti-patterns
- Treating nested students as independent (single-level OLS on clustered data)
- p-values and stars with no effect size, CI, or educational interpretation
- Reporting a total intervention effect with no test of the theorized mechanism
- Selective reporting of conditions, measures, or exclusions (undisclosed flexibility)
- Ad hoc deletion of missing data with no principled method or robustness check
Output format
【Model】multilevel / SEM / growth — nesting respected? [Y/N]
【Main result】effect size + CI + educational interpretation
【Mechanism】mediation/moderation tested as hypothesized? [Y/N/NA]
【Disclosure】N-determination + all exclusions/attrition + all measures (JARS)? [Y/N]
【Confirmatory vs exploratory】clearly separated? [Y/N]
【Reproducible】scripts + codebook + missing-data method? [Y/N]
【Next】jedpsych-tables-figures
Supplementary resources
Source: brycewang-stanford/Awesome-Journal-Skills → Journal-of-Educational-Psychology-Skills/skills/jedpsych-data-analysis/SKILL.md
1---2name: jedpsych-data-analysis3description: Use when analyzing and reporting results for a Journal of Educational Psychology manuscript. JEP expects analyses that respect nesting (multilevel/SEM/growth models), report educationally meaningful effect sizes with confidence intervals, test mechanisms (mediation/moderation), and follow JARS with full disclosure. Guides analysis norms; it does not fabricate results.4---567# Data Analysis (jedpsych-data-analysis)89The Journal of Educational Psychology holds analyses to the standards of a rigorous psychological research10journal *operating in nested educational settings*. The recurring requirements are: **model the nesting**11(students in classes in schools), report **effect sizes with confidence intervals** that are12**educationally interpretable**, test the **mechanism** (mediation/moderation), and disclose fully under13**JARS**. Analysis scripts and data are expected to be shareable and reproducible.1415## When to trigger1617- Running and reporting the main and supporting analyses18- A reviewer asked for multilevel modeling, effect sizes, mechanism tests, or disclosure19- Reconciling preregistered analyses with exploratory follow-ups20- Preparing analysis scripts and a codebook for deposit2122## Reporting norms JEP expects23241. **Respect the nesting.** Use multilevel (hierarchical linear) models, SEM, or growth models that25 account for students nested in classrooms/schools. Cluster-robust or random-effects inference is26 expected; ignoring clustering deflates standard errors and is a standard JEP rejection reason.272. **Educationally meaningful effect sizes + uncertainty.** Report a standardized effect (e.g., Hedges's28 *g*, a multilevel *d*, R²/variance explained, or a growth-rate difference) **with a confidence29 interval**, and interpret it in learning terms (e.g., months of progress, percentile shift) — not just30 p-values and stars.313. **Test the mechanism.** JEP is theory-driven: where the hypothesis includes a learning/motivational32 process, fit the mediation (with appropriate multilevel mediation methods) or moderation, not only the33 total effect.344. **Full disclosure (JARS).** Report how sample size was determined, all conditions and measures, all35 exclusions/attrition (with reasons and counts), missing-data handling (e.g., FIML/multiple imputation),36 and model specification. Confirmatory vs. exploratory must be clearly separated.375. **Appropriate inference.** Justify the model; report assumptions/diagnostics and fit indices for SEM;38 correct for multiple comparisons across many outcomes; consider robustness to alternative specifications.3940## Robustness and missing data4142- Show the result survives reasonable alternative specifications (covariate sets, model form, with/without43 exclusions). Handle attrition and missingness with principled methods (FIML, MI) and report rates by arm.4445## Worked micro-example (illustrative numbers)4647A preregistered cluster-randomized reading-comprehension trial (48 classrooms, ~1,100 students). The48confirmatory analysis is a two-level model with a pretest covariate and a preregistered mediation test.4950```51Confirmatory (preregistered) — primary effect52 Two-level model (students within classrooms), pretest-adjusted:53 classroom-level treatment effect on transfer comprehension54 g = 0.23, 95% CI [0.06, 0.40]; ICC = 0.14; ~2.0 months of progress.55 Inference uses random classroom intercepts; SEs respect clustering.56Confirmatory (preregistered) — mechanism57 Multilevel mediation: monitoring gain mediates ~40% of the effect,58 indirect 95% CI [0.02, 0.13] (excludes 0).59Sensitivity: holds with/without the preregistered attrition exclusions60 (g 0.23 → 0.21), and under FIML for missing posttests.61Exploratory (labeled): larger effect for initially low-comprehension62 readers (ATI); reported as exploratory, flagged for future confirmation.63```6465Why this passes JEP scrutiny: the model respects nesting; the effect carries a CI *and* an educational66interpretation; the mechanism is tested, not asserted; the sensitivity line pre-empts the "fragile-to-67exclusions" reviewer; and the ATI is honestly demoted to exploratory.6869## Analysis-stage reviewer pushback and the venue fix7071| Reviewer pushback | What it signals here | JEP fix |72|-------------------|----------------------|---------|73| "You ignored clustering" | deflated SEs from nesting | refit a multilevel/random-effects model; report the ICC |74| "Effect size, and what does it mean for learning?" | post-reform interpretability bar | add a CI and an educational metric (months/percentile) |75| "Mechanism untested" | total effect without theory | fit the preregistered multilevel mediation/moderation |76| "Which analyses were preregistered?" | forking-paths suspicion | give the disclosure table; relabel post hoc as exploratory |77| "How was attrition handled?" | missing-data validity | report rates by arm; use FIML/MI; show robustness |7879## Calibration anchors8081- One well-powered, properly nested effect with a tight CI and a clear educational interpretation beats a82 pile of stars from a model that treated students as independent — the latter is a routine JEP reject.83- Prefer estimation language ("the intervention raised transfer comprehension by g = 0.23, ~2 months of84 progress, 95% CI [...]") to dichotomous "significant/not."85- Mechanism evidence is what makes the paper educational *psychology* rather than evaluation; budget the86 mediation/moderation test as a first-class result, not an afterthought.8788## Execution bridge (StatsPAI / Stata MCP)8990Run the battery, don't just enumerate it. Full map:91[`execution-with-mcp`](../../../shared-resources/empirical-methods/execution-with-mcp.md). JEdPsych mixes field/lab experiments and observational school data; multilevel (student-in-class-in-school) inference and many-outcome corrections matter most.9293- **Many outcomes / specifications:** `romano_wolf` (step-down FWER) or94 `benjamini_hochberg` — report the adjusted threshold.95- **OVB sensitivity:** `oster_delta` / `sensemakr`.96- **Inference:** `wild_cluster_bootstrap` (few clusters), `twoway_cluster` / `conley`;97 multilevel data → cluster at the right level.98- **Re-fit off one handle:** `audit_result(result_id)` lists the missing checks and the99 exact `suggest_function` for each.100- **Exhibits:** `etable` / `did_summary_to_latex` from the handle — no retyped numbers.101102Keep the decisive checks in the body and the exhaustive battery in the supplement. See103the executed chain in the [JF execution walkthrough](../../../Journal-of-Finance-Skills/resources/worked-examples/02-execution-walkthrough.md).104## Anti-patterns105106- Treating nested students as independent (single-level OLS on clustered data)107- p-values and stars with no effect size, CI, or educational interpretation108- Reporting a total intervention effect with no test of the theorized mechanism109- Selective reporting of conditions, measures, or exclusions (undisclosed flexibility)110- Ad hoc deletion of missing data with no principled method or robustness check111112## Output format113114```115【Model】multilevel / SEM / growth — nesting respected? [Y/N]116【Main result】effect size + CI + educational interpretation117【Mechanism】mediation/moderation tested as hypothesized? [Y/N/NA]118【Disclosure】N-determination + all exclusions/attrition + all measures (JARS)? [Y/N]119【Confirmatory vs exploratory】clearly separated? [Y/N]120【Reproducible】scripts + codebook + missing-data method? [Y/N]121【Next】jedpsych-tables-figures122```123124## Supplementary resources125126- [`../../resources/external_tools.md`](../../resources/external_tools.md) — `lme4`/`nlme`, `lavaan`/Mplus, `mediation`, `metafor`, `effectsize`, missing-data tools127- [`../../resources/official-source-map.md`](../../resources/official-source-map.md) — JARS statistical and disclosure requirements128129---130131**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `Journal-of-Educational-Psychology-Skills/skills/jedpsych-data-analysis/SKILL.md`
thedixitjain