Arsoc Transparency And Reproducibility

Transparency & Reproducibility for a Review (arsoc-transparency-and-reproducibility)

When to trigger

• The synthesis is settled and you must show how the literature was searched and selected
• The review reports a quantitative synthesis (meta-analysis, meta-regression, citation analysis) of its own
• A Committee member or referee is likely to ask "how did you decide what to include?"
• You want the coverage account to pre-empt the predictable "you omit literature X" pushback

What "transparency" means in a review with no primary data

An ARSoc review reports no new data of its own, so the usual replication-package machinery (raw data, estimation code, a reproducibility README of your analysis) does not apply to the review's claims about the world. Transparency instead bites on two things you do produce:

1. The coverage account — a documented, near-PRISMA-style record of how the literature was searched, screened, and selected. This is what makes "comprehensive" a verifiable claim rather than a boast.
2. Any analysis the review itself runs — if the review contributes a meta-analysis, a citation/bibliometric analysis, or a re-coded dataset of studies, that artefact must be reproducible and depositable like any quantitative work.

Distinguish the two clearly: you are not defending an identification strategy of your own (there is none); you are documenting how you found and weighed others' work, and reproducing any synthesis you actually computed.

The coverage account (near-PRISMA, adapted for sociology)

Carry forward the saturation log from arsoc-literature-synthesis into a brief, sharable record:

Sociology reviews are rarely strict PRISMA systematic reviews, but a near-PRISMA narrative of the search makes the coverage defensible and is increasingly expected. Confirm whether the topic warrants a formal PRISMA flow on the author pages (检索于 2026-06；以官网为准).

When the review runs its own analysis

If the review contributes a meta-analysis or bibliometric study:

• Deposit data + code in a public repository (OSF, Dataverse, Zenodo) so the synthesis is reproducible; cite the DOI.
• State the protocol: inclusion criteria, effect-size extraction, coding rules, weighting, heterogeneity and publication-bias diagnostics.
• Pool only commensurable studies — never reduce qualitative or theoretical work to a spurious effect size (see arsoc-tables-figures).
• Make the coding reproducible: a second coder could re-derive your study-level codes from your codebook.

Checklist

• Coverage account documented (databases, dates, terms, inclusion/exclusion, saturation, counts)
• Search captured qualitative, computational, demographic, and theoretical modes, not only quant
• Near-PRISMA narrative (or formal flow if warranted) prepared; format confirmed on author pages (volatile)
• Any meta-analysis/bibliometric analysis: data + code deposited with a citable DOI
• Synthesis protocol stated (inclusion, extraction, coding rules, weighting, bias diagnostics)
• Study-level coding reproducible from a documented codebook
• Only commensurable studies pooled into any quantitative synthesis
• Coverage account ready to pre-empt "why did you omit X?" at review

Anti-patterns

• Claiming "comprehensive coverage" with no documented search to back it
• Importing primary-research replication machinery for claims the review does not itself estimate
• Treating a narrative review as exempt from any coverage account (the gap referees probe)
• Running a meta-analysis with no deposited data/code or no stated protocol
• Pooling incommensurable findings (qualitative, theoretical) into a single effect size
• Documenting only the quantitative search and silently dropping the qualitative/theoretical sweep

Output format

【Coverage account】databases + dates + terms + inclusion/exclusion + saturation + counts documented? Y/N
【Mode coverage】qual / computational / demographic / theoretical search documented? Y/N
【PRISMA】near-PRISMA narrative (or formal flow) prepared; format confirmed? Y/N · 待核实
【Own analysis】meta-analysis/bibliometrics deposited with DOI + protocol? Y/N · n/a
【Coding reproducibility】codebook lets a second coder re-derive codes? Y/N · n/a
【Pre-empt】coverage account answers "why omit X?" before review? Y/N
【Next step】→ arsoc-editor-strategy (scope + volume timeline) → arsoc-submission

Source: brycewang-stanford/Awesome-Journal-Skills → Annual-Review-of-Sociology-Skills/skills/arsoc-transparency-and-reproducibility/SKILL.md