Transparency & Reproducibility (revedres-transparency-and-reproducibility)
When to trigger
- The analysis is done and you must make the review auditable and re-runnable
- Coding reliability has not been quantified or reported
- You have not decided what search files, codebook, data, or scripts to share
- A reviewer is likely to ask "could someone reproduce this synthesis from what you report?"
Why transparency IS the contribution at RER
Because RER reviews are submitted and peer-reviewed (not invited), a reader cannot take the synthesis on the author's authority — they must be able to check it. The credibility of a systematic review or meta-analysis is exactly the credibility of its process documentation: the search, the screening, the coding, and the analysis must be reported so completely that an independent reader could, in principle, re-run them and land in the same place. This is not an appendix afterthought; it is the spine of the review's claim to be systematic.
Reporting standards RER points to
RER directs authors of systematic reviews and meta-analyses to specific reporting standards (检索于 2026-06;以官网为准):
- PRISMA — the systematic-review/meta-analysis reporting checklist and flow diagram. Report against every applicable item; include the flow diagram (see
revedres-tables-figures).
- APA "Reporting Standards for Research in Psychology" (the JARS family, incl. MARS for meta-analyses) — RER's submission guidance cites the American Psychologist reporting-standards article. Use MARS to ensure the meta-analysis reports its eligibility, search, coding, effect-size metric, model, and heterogeneity/bias analyses completely.
- PRISMA-P for the protocol (see
revedres-proposal-and-commissioning); PROSPERO/OSF registration cited.
Coding reliability: report it, don't assert it
- Double-code a defined share of studies (or all), independently.
- Quantify agreement with the appropriate statistic — Cohen's/Fleiss' κ for categorical codes, ICC for continuous extractions — and report the value, not just "two coders agreed."
- State conflict resolution (consensus, third coder) and re-check after disagreements.
- Provide the codebook so a reader knows exactly what was coded and how.
Open materials: make the synthesis re-runnable
- Share the search artifacts: full strings per database, dates, and the screening decisions/PRISMA counts.
- Share the coding dataset and codebook (de-identified as needed) on OSF or a repository, with a DOI.
- Share the analysis code (R
metafor/meta, Stata meta, or Python) so the pooled estimates, moderators, and bias diagnostics are reproducible from the shared data.
- Distinguish preregistered from exploratory analyses, and flag/justify every deviation from the protocol.
- Note: RER publishes no original empirical data of yours; the open-materials obligation is on the review's own search, coding, and synthesis — your reproducible contribution.
Checklist
Anti-patterns
- "Two coders agreed" with no κ/ICC value (unauditable reliability)
- A meta-analysis whose model, weights, or effect-size metric a reader cannot reconstruct
- Search strings paraphrased, not reported verbatim per database
- Silent deviations from the registered protocol
- Treating open materials as optional polish rather than the basis of the systematic claim
- Sharing nothing because "it's only a review" — the review's own process is the data
Output format
【PRISMA】checklist + flow complete? Y/N
【MARS/JARS】meta-analysis reporting standards walked? Y/N
【Registration】PROSPERO/OSF cited; deviations flagged? Y/N
【Coder reliability】κ/ICC value reported = <value>; conflict resolution stated? Y/N
【Codebook】included? Y/N
【Search artifacts】strings + dates + screening counts reported in full? Y/N
【Open materials】dataset + code shared (DOI)? Y/N
【Preregistered vs exploratory】distinguished? Y/N
【Next step】→ revedres-editor-strategy (scope negotiation + reviewer expectations)
1---2name: revedres-transparency-and-reproducibility3description: Use when settling coding reliability, open materials, PRISMA/MARS reporting, and a reproducible meta-analysis for a Review of Educational Research (RER) manuscript. Makes the review auditable and re-runnable; it does not run the substantive robustness analyses (revedres-comprehensiveness-and-balance) or design the exhibits (revedres-tables-figures).4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/revedres-transparency-and-reproducibility6---78# Transparency & Reproducibility (revedres-transparency-and-reproducibility)910## When to trigger1112- The analysis is done and you must make the review auditable and re-runnable13- Coding reliability has not been quantified or reported14- You have not decided what search files, codebook, data, or scripts to share15- A reviewer is likely to ask "could someone reproduce this synthesis from what you report?"1617## Why transparency IS the contribution at RER1819Because RER reviews are **submitted and peer-reviewed** (not invited), a reader cannot take the synthesis on the author's authority — they must be able to **check it**. The credibility of a systematic review or meta-analysis is exactly the credibility of its **process documentation**: the search, the screening, the coding, and the analysis must be reported so completely that an independent reader could, in principle, re-run them and land in the same place. This is not an appendix afterthought; it is the spine of the review's claim to be *systematic*.2021## Reporting standards RER points to2223RER directs authors of systematic reviews and meta-analyses to specific reporting standards (检索于 2026-06;以官网为准):2425- **PRISMA** — the systematic-review/meta-analysis reporting checklist and flow diagram. Report against every applicable item; include the flow diagram (see `revedres-tables-figures`).26- **APA "Reporting Standards for Research in Psychology"** (the JARS family, incl. **MARS** for meta-analyses) — RER's submission guidance cites the *American Psychologist* reporting-standards article. Use MARS to ensure the meta-analysis reports its eligibility, search, coding, effect-size metric, model, and heterogeneity/bias analyses completely.27- **PRISMA-P** for the protocol (see `revedres-proposal-and-commissioning`); **PROSPERO/OSF** registration cited.2829## Coding reliability: report it, don't assert it30311. **Double-code a defined share** of studies (or all), independently.322. **Quantify agreement** with the appropriate statistic — Cohen's/Fleiss' κ for categorical codes, ICC for continuous extractions — and **report the value**, not just "two coders agreed."333. **State conflict resolution** (consensus, third coder) and re-check after disagreements.344. Provide the **codebook** so a reader knows exactly what was coded and how.3536## Open materials: make the synthesis re-runnable3738- **Share the search artifacts**: full strings per database, dates, and the screening decisions/PRISMA counts.39- **Share the coding dataset and codebook** (de-identified as needed) on OSF or a repository, with a DOI.40- **Share the analysis code** (R `metafor`/`meta`, Stata `meta`, or Python) so the pooled estimates, moderators, and bias diagnostics are reproducible from the shared data.41- **Distinguish preregistered from exploratory** analyses, and flag/justify every deviation from the protocol.42- Note: RER publishes no original empirical data of *yours*; the open-materials obligation is on the **review's own** search, coding, and synthesis — your reproducible contribution.4344## Checklist4546- [ ] PRISMA checklist + flow diagram completed (systematic/meta)47- [ ] MARS / APA reporting standards walked for a meta-analysis48- [ ] Protocol registration (PROSPERO/OSF) cited; deviations flagged and justified49- [ ] Inter-coder reliability quantified (κ/ICC) and reported, not asserted50- [ ] Codebook included; conflict-resolution procedure stated51- [ ] Search strings, dates, and screening counts reported in full52- [ ] Coding dataset + analysis code shared (OSF/repository, with DOI) where possible53- [ ] Preregistered vs. exploratory analyses clearly distinguished5455## Anti-patterns5657- "Two coders agreed" with no κ/ICC value (unauditable reliability)58- A meta-analysis whose model, weights, or effect-size metric a reader cannot reconstruct59- Search strings paraphrased, not reported verbatim per database60- Silent deviations from the registered protocol61- Treating open materials as optional polish rather than the basis of the systematic claim62- Sharing nothing because "it's only a review" — the review's own process is the data6364## Output format6566```67【PRISMA】checklist + flow complete? Y/N68【MARS/JARS】meta-analysis reporting standards walked? Y/N69【Registration】PROSPERO/OSF cited; deviations flagged? Y/N70【Coder reliability】κ/ICC value reported = <value>; conflict resolution stated? Y/N71【Codebook】included? Y/N72【Search artifacts】strings + dates + screening counts reported in full? Y/N73【Open materials】dataset + code shared (DOI)? Y/N74【Preregistered vs exploratory】distinguished? Y/N75【Next step】→ revedres-editor-strategy (scope negotiation + reviewer expectations)76```