Revedres Transparency And Reproducibility

Transparency & Reproducibility (revedres-transparency-and-reproducibility)

When to trigger

• The analysis is done and you must make the review auditable and re-runnable
• Coding reliability has not been quantified or reported
• You have not decided what search files, codebook, data, or scripts to share
• A reviewer is likely to ask "could someone reproduce this synthesis from what you report?"

Why transparency IS the contribution at RER

Because RER reviews are submitted and peer-reviewed (not invited), a reader cannot take the synthesis on the author's authority — they must be able to check it. The credibility of a systematic review or meta-analysis is exactly the credibility of its process documentation: the search, the screening, the coding, and the analysis must be reported so completely that an independent reader could, in principle, re-run them and land in the same place. This is not an appendix afterthought; it is the spine of the review's claim to be systematic.

Reporting standards RER points to

RER directs authors of systematic reviews and meta-analyses to specific reporting standards (检索于 2026-06；以官网为准):

• PRISMA — the systematic-review/meta-analysis reporting checklist and flow diagram. Report against every applicable item; include the flow diagram (see revedres-tables-figures).
• APA "Reporting Standards for Research in Psychology" (the JARS family, incl. MARS for meta-analyses) — RER's submission guidance cites the American Psychologist reporting-standards article. Use MARS to ensure the meta-analysis reports its eligibility, search, coding, effect-size metric, model, and heterogeneity/bias analyses completely.
• PRISMA-P for the protocol (see revedres-proposal-and-commissioning); PROSPERO/OSF registration cited.

Coding reliability: report it, don't assert it

1. Double-code a defined share of studies (or all), independently.
2. Quantify agreement with the appropriate statistic — Cohen's/Fleiss' κ for categorical codes, ICC for continuous extractions — and report the value, not just "two coders agreed."
3. State conflict resolution (consensus, third coder) and re-check after disagreements.
4. Provide the codebook so a reader knows exactly what was coded and how.

Open materials: make the synthesis re-runnable

• Share the search artifacts: full strings per database, dates, and the screening decisions/PRISMA counts.
• Share the coding dataset and codebook (de-identified as needed) on OSF or a repository, with a DOI.
• Share the analysis code (R metafor/meta, Stata meta, or Python) so the pooled estimates, moderators, and bias diagnostics are reproducible from the shared data.
• Distinguish preregistered from exploratory analyses, and flag/justify every deviation from the protocol.
• Note: RER publishes no original empirical data of yours; the open-materials obligation is on the review's own search, coding, and synthesis — your reproducible contribution.

Checklist

• PRISMA checklist + flow diagram completed (systematic/meta)
• MARS / APA reporting standards walked for a meta-analysis
• Protocol registration (PROSPERO/OSF) cited; deviations flagged and justified
• Inter-coder reliability quantified (κ/ICC) and reported, not asserted
• Codebook included; conflict-resolution procedure stated
• Search strings, dates, and screening counts reported in full
• Coding dataset + analysis code shared (OSF/repository, with DOI) where possible
• Preregistered vs. exploratory analyses clearly distinguished

Anti-patterns

• "Two coders agreed" with no κ/ICC value (unauditable reliability)
• A meta-analysis whose model, weights, or effect-size metric a reader cannot reconstruct
• Search strings paraphrased, not reported verbatim per database
• Silent deviations from the registered protocol
• Treating open materials as optional polish rather than the basis of the systematic claim
• Sharing nothing because "it's only a review" — the review's own process is the data

Output format

【PRISMA】checklist + flow complete? Y/N
【MARS/JARS】meta-analysis reporting standards walked? Y/N
【Registration】PROSPERO/OSF cited; deviations flagged? Y/N
【Coder reliability】κ/ICC value reported = <value>; conflict resolution stated? Y/N
【Codebook】included? Y/N
【Search artifacts】strings + dates + screening counts reported in full? Y/N
【Open materials】dataset + code shared (DOI)? Y/N
【Preregistered vs exploratory】distinguished? Y/N
【Next step】→ revedres-editor-strategy (scope negotiation + reviewer expectations)

Source: brycewang-stanford/Awesome-Journal-Skills → Review-of-Educational-Research-Skills/skills/revedres-transparency-and-reproducibility/SKILL.md