Vambrocop Umbrella Review Skeptic
Use this skill when the evidence base already contains reviews or meta-analyses and the user wants to synthesize the syntheses.
Core Principle
Do not simply average meta-analyses. Existing reviews often reuse the same primary studies, use different inclusion rules, and report non-comparable pooled effects.
Intake
Identify:
- included reviews/meta-analyses;
- review questions;
- search dates;
- primary studies included in each review;
- pooled effects and metrics;
- quality assessment method;
- overlap among reviews;
- whether a second-order statistical synthesis is planned.
Load:
references/overlap-and-quality.mdfor overlap and quality checks.references/second-order-decision-rules.mdwhen deciding whether review-level pooling is defensible.references/agricultural-diversification-second-order-meta.mdfor long-term agricultural diversification, ecosystem-service, profitability, and yield trade-off synthesis.
Workflow
- Classify the project: umbrella review, review of reviews, or second-order meta-analysis.
- Extract review-level characteristics.
- Build a primary-study by review matrix.
- Assess overlap and duplicate evidence.
- Assess review quality and search currency.
- Compare effect metrics and inclusion criteria.
- Apply AMSTAR 2 / ROBIS-style review appraisal logic where appropriate.
- Decide: narrative umbrella synthesis, evidence map, or second-order pooling.
- If pooling, state dependence assumptions and model choice.
- Report discordance and certainty.
Use templates/overlap-matrix.md for overlap tracking.
Use templates/second-order-temporal-tradeoff-audit.md and templates/review-level-effect-schema.csv for review-level temporal meta-regression and yield-service trade-off audits.
Use templates/second-order-r-package-ledger.csv when extracting software packages and model roles from a second-order meta-analysis.
Use templates/osf-r-dependency-inventory.csv when auditing OSF/GitHub R scripts beyond the packages named in the paper.
Use templates/second-order-quality-scorecard.csv, templates/second-order-model-spec-ledger.csv, and templates/agricultural-diversification-taxonomy.csv when adapting agricultural diversification second-order meta-analysis protocols.
Use templates/second-order-peer-review-readiness-checklist.csv before submission or revision to anticipate reviewer concerns about second-order environmental meta-analyses.
Output Modes
Umbrella Audit
Review set:
Overlap risk:
Quality concerns:
Effect comparability:
Can pool statistically?
Preferred synthesis:
Main caution:
Second-Order Meta Plan
Review-level effect:
Primary-study overlap:
Dependence handling:
Quality weighting/sensitivity:
Model:
Interpretation limits:
Temporal Trade-Off Audit
Use when the second-order synthesis asks how effects change over years or decades.
Review-level evidence base:
Duration moderator:
Outcome families:
Review-level weights:
Random effects:
Primary-study overlap:
Trade-off categories:
Long-term research gaps:
Policy interpretation limits:
Guardrails
- Do not double count the same primary studies.
- Do not pool reviews with incompatible effect metrics without justification.
- Do not treat many reviews as many independent bodies of evidence.
- Do not ignore older reviews being superseded by newer reviews.
- Do not call a narrative umbrella review a second-order meta-analysis.
- Do not assign a simple total score to review quality when the appraisal tool warns against it.
- Do not interpret review-level duration trends as primary-study causal effects without checking overlap, review scope, and the underlying time data.
- Do not collapse yield, profitability, biodiversity, soil quality, and climate mitigation into one sustainability score unless the weighting rule is explicit.