yogsoth-ai Heterogeneity Investigation Strategy
Design a protocol to investigate and explain between-study heterogeneity — why studies of the same question reach different conclusions.
Purpose
When a meta-analysis reveals substantial heterogeneity (I2 > 50%, significant Q-test, large tau2), this strategy designs the investigation protocol: subgroup analyses, meta-regression, moderator identification, and outlier diagnostics. Produces the investigation plan, not the computation.
Budget
| Resource | Floor | Target |
|---|---|---|
| Studies identified | 20 | 30 |
| Effect sizes extracted | 20 | 30 |
| Web searches | 35 | 50 |
| Moderator candidates | 5 | 10+ |
| Quality assessments | 15 | 30 |
Budget gate: cannot exit until 80% of floor met.
State Ledger
<HARD-GATE>
| Metric | Current | Floor | Target | Status |
|--------|---------|-------|--------|--------|
| Studies found | 0 | 20 | 30 | BLOCKED |
| Effect sizes planned | 0 | 20 | 30 | BLOCKED |
| Web searches done | 0 | 35 | 50 | BLOCKED |
| Moderators identified | 0 | 5 | 10+ | BLOCKED |
| Quality assessed | 0 | 15 | 30 | BLOCKED |
</HARD-GATE>
Available Tactics
| Tactic | When to Use |
|---|---|
| effect-size-extraction | Extract effect sizes with full study characteristics |
| quality-assessment-protocol | Assess whether quality explains heterogeneity |
| evidence-synthesis-planning | Plan subgroup and meta-regression models |
Available SOPs
| SOP | When to Use |
|---|---|
| pico-formulation | Frame the heterogeneity question |
| inclusion-criteria-design | Broad inclusion to capture variation |
| effect-size-planning | Standardize for comparability |
| data-extraction-form | Rich moderator variable extraction |
| risk-of-bias-assessment | RoB as potential moderator |
| heterogeneity-source-analysis | Core SOP — classify heterogeneity sources |
| sensitivity-analysis-design | Outlier removal, influence diagnostics |
| publication-bias-assessment | Bias as heterogeneity source |
| meta-analysis-synthesis | Final investigation protocol |
Execution Guidance
- Frame — Run
pico-formulationemphasizing variation in P/I/C/O - Scope — Run
inclusion-criteria-designwith broad criteria (capture variation) - Search — Systematic search + web research on known moderators
- Extract — Use
effect-size-extractionwith rich study-level covariates - Hypothesize — Run
heterogeneity-source-analysisto generate moderator hypotheses - Assess — Use
quality-assessment-protocol(RoB as moderator) - Plan — Use
evidence-synthesis-planningfor subgroup + meta-regression - Synthesize — Run
meta-analysis-synthesisfor investigation protocol
Web searches focus on domain knowledge about why results might differ (methodological, clinical, statistical heterogeneity).
Output Format
protocol:
question: [Why do studies of X reach different conclusions?]
heterogeneity_metrics: [I2, tau2, Q-test, prediction interval]
moderator_candidates:
clinical: [population, intervention details, outcome timing]
methodological: [study design, RoB, measurement tools]
statistical: [effect size type, analysis method, sample size]
investigation_plan:
subgroup_analyses: [categorical moderators]
meta_regression: [continuous moderators]
outlier_diagnostics: [influence analysis, Baujat plot]
sensitivity: [leave-one-out, cumulative by quality]
a_priori_hypotheses: [pre-specified moderator hypotheses]
multiple_testing: [correction strategy]
reporting: PRISMA-2020 + heterogeneity reporting guidelines
Available Tactics
Optional, no fixed order; the final leaf is always a sop.
| Tactic | When to use |
|---|---|
| effect-size-extraction | Systematically extract effect sizes and conditions from papers for meta-analytic synthesis |
| evidence-synthesis-planning | Plan the statistical synthesis approach — model selection, heterogeneity strategy, and reporting |
| quality-assessment-protocol | Methodological quality and bias risk assessment of included studies using validated tools |
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP | When to use |
|---|---|
| data-extraction-form | Design structured data extraction form for systematic meta-analysis data collection |
| effect-size-planning | Determine effect size types and calculation methods for meta-analytic synthesis |
| heterogeneity-source-analysis | Identify and classify sources of between-study heterogeneity (clinical, methodological, statistical) |
| inclusion-criteria-design | Define inclusion/exclusion criteria for systematic study selection in meta-analysis |
| meta-analysis-synthesis | Produce final meta-analysis protocol document assembling all planning outputs into PRISMA-compliant protocol |
| pico-formulation | Construct PICO/PECO framework for the meta-analysis research question |
| publication-bias-assessment | Plan funnel plots, Egger's test, trim-and-fill, p-curve, and selection model analyses for publication bias |
| risk-of-bias-assessment | Assess methodological bias using RoB2, PROBAST, or QUADAS-2 validated tools |
| sensitivity-analysis-design | Design leave-one-out, influence diagnostics, subgroup analyses, and robustness checks |