Comprehensiveness & Balance (revedres-comprehensiveness-and-balance)
When to trigger
- The corpus and framework exist, but you have not stress-tested coverage or robustness
- A pooled effect is reported without heterogeneity, bias, or sensitivity analysis
- You worry a reviewer will name an omitted study, school, or contradictory finding
- Conflicting studies are being tallied rather than weighed by design quality
Comprehensiveness: prove there are no holes
RER asks for comprehensive reviews; the burden is on you to make exhaustiveness provable, not asserted.
- Saturation evidence. Show that the search reached the point where new searches stopped yielding eligible studies (from
revedres-literature-synthesis).
- Named-omission test. Could an informed reviewer name an important study, research group, or adjacent literature you left out? If yes, include it or justify the boundary explicitly.
- Grey-literature & language reach. Excluding dissertations, reports, or non-English work narrows scope and biases effects — acknowledge and, where feasible, widen.
Balance: weigh evidence, never vote-count
The amateur move is vote-counting — tallying significant vs. null studies. The RER standard is to weigh evidence by what each study measures and how credibly.
- Risk-of-bias appraisal. Apply the a-priori tool to every included study; let credibility, not count, drive emphasis. A handful of well-identified studies can outweigh many weak ones.
- Estimand discipline. Reconcile conflicting findings by asking whether studies estimate the same object (population, construct, horizon, comparison). Apparent contradictions often dissolve.
- Steelman rival perspectives. State each theoretical or methodological camp at its strongest before its limits; flag — and bracket — your own program so emphasis is identity-blind.
Robustness (meta-analysis): make the number survive scrutiny
A pooled effect is a claim, not a fact, until you show it is not an artifact.
| Probe |
What it guards against |
| Heterogeneity (Q, I², τ²) |
reporting one number for a mix of different effects |
| Moderator analysis |
masking real variation the framework should explain |
| Publication-bias diagnostics (funnel, Egger, trim-and-fill, p-curve/selection models) |
an inflated effect from missing null results |
| Sensitivity analysis (leave-one-out, influence, alternative models) |
a result driven by one study or one modeling choice |
| Dependent-effects handling (multilevel / robust variance) |
false precision from multiple effects per sample |
For a narrative synthesis, the analogues are: confidence in the body of evidence (e.g. a GRADE-style judgment), explicit handling of conflicting findings, and a sensitivity check on which conclusions survive dropping the weakest studies.
Checklist
Anti-patterns
- Vote-counting significant vs. null studies as if each carries equal weight
- A single pooled effect with no I²/τ², no moderators, and no publication-bias check
- Ignoring dependent effect sizes, manufacturing false precision
- Excluding grey literature silently, then reporting an upward-biased effect
- Caricaturing the camp the author disagrees with instead of steelmanning it
- Treating "I found a lot of studies" as proof of comprehensiveness without saturation evidence
Output format
【Saturation】documented? Y/N — named-omission test passed? Y/N
【Grey lit / language】exclusions + bias implication stated? Y/N
【Risk of bias】appraised for all studies; emphasis credibility-weighted? Y/N
【Conflict handling】reconciled by estimand/design (not vote-count)? Y/N
【Heterogeneity】I²/τ² + moderators reported? Y/N (meta) | strength-of-evidence judged (narrative)
【Publication bias】funnel/Egger/trim-fill/p-curve run? Y/N
【Sensitivity】leave-one-out / alt models / dependent-effects model? Y/N
【Next step】→ revedres-tables-figures (PRISMA flow, forest/funnel, coding tables)
1---2name: revedres-comprehensiveness-and-balance3description: Use when testing exhaustiveness, risk-of-bias, heterogeneity, and sensitivity for a Review of Educational Research (RER) review or meta-analysis. Hardens the synthesis against omission and fragility; it does not build the conceptual spine (revedres-organizing-framework) or settle coding reliability and open materials (revedres-transparency-and-reproducibility).4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/revedres-comprehensiveness-and-balance6---78# Comprehensiveness & Balance (revedres-comprehensiveness-and-balance)910## When to trigger1112- The corpus and framework exist, but you have not stress-tested coverage or robustness13- A pooled effect is reported without heterogeneity, bias, or sensitivity analysis14- You worry a reviewer will name an omitted study, school, or contradictory finding15- Conflicting studies are being tallied rather than weighed by design quality1617## Comprehensiveness: prove there are no holes1819RER asks for **comprehensive** reviews; the burden is on you to make exhaustiveness *provable*, not asserted.20211. **Saturation evidence.** Show that the search reached the point where new searches stopped yielding eligible studies (from `revedres-literature-synthesis`).222. **Named-omission test.** Could an informed reviewer name an important study, research group, or adjacent literature you left out? If yes, include it or justify the boundary explicitly.233. **Grey-literature & language reach.** Excluding dissertations, reports, or non-English work narrows scope and biases effects — acknowledge and, where feasible, widen.2425## Balance: weigh evidence, never vote-count2627The amateur move is **vote-counting** — tallying significant vs. null studies. The RER standard is to **weigh** evidence by what each study measures and how credibly.28291. **Risk-of-bias appraisal.** Apply the a-priori tool to every included study; let credibility, not count, drive emphasis. A handful of well-identified studies can outweigh many weak ones.302. **Estimand discipline.** Reconcile conflicting findings by asking whether studies estimate the *same object* (population, construct, horizon, comparison). Apparent contradictions often dissolve.313. **Steelman rival perspectives.** State each theoretical or methodological camp at its strongest before its limits; flag — and bracket — your own program so emphasis is identity-blind.3233## Robustness (meta-analysis): make the number survive scrutiny3435A pooled effect is a claim, not a fact, until you show it is not an artifact.3637| Probe | What it guards against |38|---|---|39| **Heterogeneity** (Q, I², τ²) | reporting one number for a mix of different effects |40| **Moderator analysis** | masking real variation the framework should explain |41| **Publication-bias diagnostics** (funnel, Egger, trim-and-fill, p-curve/selection models) | an inflated effect from missing null results |42| **Sensitivity analysis** (leave-one-out, influence, alternative models) | a result driven by one study or one modeling choice |43| **Dependent-effects handling** (multilevel / robust variance) | false precision from multiple effects per sample |4445For a narrative synthesis, the analogues are: confidence in the body of evidence (e.g. a GRADE-style judgment), explicit handling of conflicting findings, and a sensitivity check on which conclusions survive dropping the weakest studies.4647## Checklist4849- [ ] Saturation documented; no eligible study/database a reviewer could name as missing50- [ ] Grey-literature/language exclusions acknowledged with their bias implications51- [ ] Risk-of-bias appraised for every study; emphasis tracks credibility, not count52- [ ] Conflicting findings reconciled by estimand + design, not tallied53- [ ] Rival camps steelmanned; author's own work bracketed for identity-blind emphasis54- [ ] (Meta) heterogeneity, moderators, publication-bias, and sensitivity all reported55- [ ] (Meta) dependent effects modeled (multilevel / robust variance), not ignored56- [ ] (Narrative) strength-of-evidence judged and a drop-the-weakest sensitivity check run5758## Anti-patterns5960- Vote-counting significant vs. null studies as if each carries equal weight61- A single pooled effect with no I²/τ², no moderators, and no publication-bias check62- Ignoring dependent effect sizes, manufacturing false precision63- Excluding grey literature silently, then reporting an upward-biased effect64- Caricaturing the camp the author disagrees with instead of steelmanning it65- Treating "I found a lot of studies" as proof of comprehensiveness without saturation evidence6667## Output format6869```70【Saturation】documented? Y/N — named-omission test passed? Y/N71【Grey lit / language】exclusions + bias implication stated? Y/N72【Risk of bias】appraised for all studies; emphasis credibility-weighted? Y/N73【Conflict handling】reconciled by estimand/design (not vote-count)? Y/N74【Heterogeneity】I²/τ² + moderators reported? Y/N (meta) | strength-of-evidence judged (narrative)75【Publication bias】funnel/Egger/trim-fill/p-curve run? Y/N76【Sensitivity】leave-one-out / alt models / dependent-effects model? Y/N77【Next step】→ revedres-tables-figures (PRISMA flow, forest/funnel, coding tables)78```