Systematic Search & Synthesis (revedres-literature-synthesis)
When to trigger
- The protocol is fixed and it is time to search the literature exhaustively
- Searching feels ad hoc; you cannot yet report a reproducible PRISMA flow
- You have hundreds of records and need a defensible screening trail
- A reviewer at RER is likely to ask "why did you omit study X / database Y?"
Search to a documented PRISMA flow, not to memory
An RER systematic review's credibility rests on a reader's belief that you found everything that meets your criteria — and can prove it. Execute the protocol, logging every number for the PRISMA flow diagram (identification → screening → eligibility → included).
- Run the registered search. Search every database in the protocol (ERIC, PsycINFO, Education Source, Web of Science, Scopus, ProQuest Dissertations) with the recorded strings, plus grey-literature and hand-searches of key journals. Record hits per source and the search date.
- Deduplicate and log. Report records identified, duplicates removed, and records screened — exact counts.
- Dual independent screening. Two screeners at title/abstract, then full-text, against the eligibility criteria; record exclusions with reasons at full-text (required by PRISMA). Report inter-rater reliability (Cohen's κ or % agreement) and how conflicts were resolved.
- Supplement to saturation. Backward (reference lists of included studies) and forward (who cites them) snowballing; ask whether new searches still surface eligible studies. Document where they stop.
- Extract into a structured dataset. Apply the codebook to every included study — this is the raw material for the framework, the tables, and the meta-analysis.
From extraction to synthesis (not summary)
Summarizing is restating each study; synthesizing is making the studies answer your question together. Maintain a coding dataset as you extract:
| Column |
What to capture |
| Study |
author–year; the included report (watch for multiple reports of one sample) |
| Sample/context |
learners, setting, grade/level, country — for moderator analysis and scope claims |
| Design |
RCT / quasi-experiment / correlational / qualitative — for risk-of-bias and weighting |
| Construct/measure |
exactly what was measured (so non-commensurable outcomes are not pooled) |
| Effect / finding |
effect size + variance (meta-analysis) or coded finding (narrative synthesis) |
| Risk of bias |
your appraisal on the a-priori tool (you do not re-run the study; you judge it) |
| Dependencies |
shared samples / multiple effects per study (drives the variance model) |
This dataset feeds the organizing framework, the forest plot and coding tables, and the even-handed treatment of conflicting evidence. You appraise the primary studies (you are the field's reviewer-of-record); you do not re-collect their data.
Education-specific search hazards
The education literature is scattered across disciplines and document types, which creates predictable holes:
- Cross-disciplinary indexing. Relevant work hides in psychology (PsycINFO), economics (EconLit/NBER), sociology, and policy databases — searching only ERIC misses it. Map your constructs to each field's vocabulary.
- Grey literature is large and consequential. Dissertations (ProQuest), technical and foundation reports, and What Works Clearinghouse / IES products carry many null and small-sample results; omitting them biases pooled effects upward.
- Terminology drift. The same construct is named differently across eras and subfields (e.g. "self-regulation" vs. "metacognition" vs. "executive function") — build a thesaurus of synonyms into the search string.
- Multiple reports of one study. Program evaluations spawn several papers on the same sample; collapse them to one unit or model the dependency, or you double-count.
Document how you handled each, so a reviewer sees the gaps were anticipated, not missed.
Checklist
Anti-patterns
- Searching from memory or one database (predictable, fatal coverage gaps at RER)
- A PRISMA diagram whose numbers do not reconcile (a red flag reviewers check)
- Single-screener inclusion with no reliability statistic
- Excluding grey literature without acknowledging the publication-bias risk it creates
- Pooling outcomes that measure different constructs into one "effect of X"
- Re-analyzing or "correcting" a primary study's raw data — you appraise, you do not re-collect
Output format
【Databases + date】<sources searched, search date>
【PRISMA counts】identified / dedup / screened / full-text / excluded-w-reasons / included
【Screening reliability】κ or % agreement; conflicts resolved by <method>
【Snowballing】backward + forward to saturation? Y/N
【Grey literature】included? Y/N — publication-bias implication noted? Y/N
【Coding dataset】codebook applied; dependent effects + shared samples flagged? Y/N
【Coverage risks】<any eligible study/database a reviewer could name as missing>
【Next step】→ revedres-organizing-framework (impose the conceptual spine on the corpus)
Source: brycewang-stanford/Awesome-Journal-Skills → Review-of-Educational-Research-Skills/skills/revedres-literature-synthesis/SKILL.md
1---2name: revedres-literature-synthesis3description: Use when executing the systematic search, screening to a PRISMA flow, and extracting data for a Review of Educational Research (RER) review or meta-analysis. Builds the documented evidence corpus; it does not impose the conceptual spine (revedres-organizing-framework) or run robustness/risk-of-bias appraisal (revedres-comprehensiveness-and-balance).4---5
6
7# Systematic Search & Synthesis (revedres-literature-synthesis)
8
9## When to trigger
10
11- The protocol is fixed and it is time to search the literature exhaustively
12- Searching feels ad hoc; you cannot yet report a reproducible PRISMA flow
13- You have hundreds of records and need a defensible screening trail
14- A reviewer at RER is likely to ask "why did you omit study X / database Y?"
15
16## Search to a documented PRISMA flow, not to memory
17
18An RER systematic review's credibility rests on a reader's belief that you found **everything that meets your criteria** — and can prove it. Execute the protocol, logging every number for the **PRISMA flow diagram** (identification → screening → eligibility → included).
19
201. **Run the registered search.** Search every database in the protocol (ERIC, PsycINFO, Education Source, Web of Science, Scopus, ProQuest Dissertations) with the recorded strings, plus grey-literature and hand-searches of key journals. Record hits per source and the search date.
212. **Deduplicate and log.** Report records identified, duplicates removed, and records screened — exact counts.
223. **Dual independent screening.** Two screeners at title/abstract, then full-text, against the eligibility criteria; record exclusions **with reasons** at full-text (required by PRISMA). Report inter-rater reliability (Cohen's κ or % agreement) and how conflicts were resolved.
234. **Supplement to saturation.** Backward (reference lists of included studies) and forward (who cites them) snowballing; ask whether new searches still surface eligible studies. Document where they stop.
245. **Extract into a structured dataset.** Apply the codebook to every included study — this is the raw material for the framework, the tables, and the meta-analysis.
25
26## From extraction to synthesis (not summary)
27
28Summarizing is restating each study; **synthesizing** is making the studies answer your question together. Maintain a coding dataset as you extract:
29
30| Column | What to capture |
31|--------|-----------------|
32| Study | author–year; the included report (watch for multiple reports of one sample) |
33| Sample/context | learners, setting, grade/level, country — for moderator analysis and scope claims |
34| Design | RCT / quasi-experiment / correlational / qualitative — for risk-of-bias and weighting |
35| Construct/measure | exactly what was measured (so non-commensurable outcomes are not pooled) |
36| Effect / finding | effect size + variance (meta-analysis) or coded finding (narrative synthesis) |
37| Risk of bias | your appraisal on the a-priori tool (you do not re-run the study; you judge it) |
38| Dependencies | shared samples / multiple effects per study (drives the variance model) |
39
40This dataset feeds the organizing framework, the forest plot and coding tables, and the even-handed treatment of conflicting evidence. You **appraise** the primary studies (you are the field's reviewer-of-record); you do **not** re-collect their data.
41
42## Education-specific search hazards
43
44The education literature is scattered across disciplines and document types, which creates predictable holes:
45
46- **Cross-disciplinary indexing.** Relevant work hides in psychology (PsycINFO), economics (EconLit/NBER), sociology, and policy databases — searching only ERIC misses it. Map your constructs to each field's vocabulary.
47- **Grey literature is large and consequential.** Dissertations (ProQuest), technical and foundation reports, and What Works Clearinghouse / IES products carry many null and small-sample results; omitting them biases pooled effects upward.
48- **Terminology drift.** The same construct is named differently across eras and subfields (e.g. "self-regulation" vs. "metacognition" vs. "executive function") — build a thesaurus of synonyms into the search string.
49- **Multiple reports of one study.** Program evaluations spawn several papers on the same sample; collapse them to one unit or model the dependency, or you double-count.
50
51Document how you handled each, so a reviewer sees the gaps were anticipated, not missed.
52
53## Checklist
54
55- [ ] Every protocol database searched with recorded strings + search date
56- [ ] Records identified / duplicates / screened / excluded-with-reasons / included all counted for PRISMA
57- [ ] Dual independent screening; inter-rater reliability reported; conflict resolution stated
58- [ ] Backward + forward snowballing run to saturation and documented
59- [ ] Grey literature / dissertations handled per protocol (and publication-bias implications noted)
60- [ ] Codebook applied uniformly; multiple-reports-of-one-sample and dependent effects flagged
61- [ ] Non-commensurable outcomes flagged (not pooled into a false common effect)
62- [ ] No eligible study or relevant database an informed reviewer could name as missing
63
64## Anti-patterns
65
66- Searching from memory or one database (predictable, fatal coverage gaps at RER)
67- A PRISMA diagram whose numbers do not reconcile (a red flag reviewers check)
68- Single-screener inclusion with no reliability statistic
69- Excluding grey literature without acknowledging the publication-bias risk it creates
70- Pooling outcomes that measure different constructs into one "effect of X"
71- Re-analyzing or "correcting" a primary study's raw data — you appraise, you do not re-collect
72
73## Output format
74
75```
76【Databases + date】<sources searched, search date>
77【PRISMA counts】identified / dedup / screened / full-text / excluded-w-reasons / included
78【Screening reliability】κ or % agreement; conflicts resolved by <method>
79【Snowballing】backward + forward to saturation? Y/N
80【Grey literature】included? Y/N — publication-bias implication noted? Y/N
81【Coding dataset】codebook applied; dependent effects + shared samples flagged? Y/N
82【Coverage risks】<any eligible study/database a reviewer could name as missing>
83【Next step】→ revedres-organizing-framework (impose the conceptual spine on the corpus)
84```
85
86---
87
88**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `Review-of-Educational-Research-Skills/skills/revedres-literature-synthesis/SKILL.md`
thedixitjain