Evidence Synthesis

Evidence Synthesis

Individual pieces of evidence are unreliable. Synthesis across sources — weighting by quality, identifying convergence and divergence, resolving conflicts — produces conclusions worth acting on. Conclusion quality is bounded by the quality of the weakest evidence it leans on most heavily.

When to Use

• Multiple sources need integration into a single conclusion
• causal-inference produced experimental findings to weigh against other data
• learning-strategy produced summaries from multiple sources
• Sources disagree and the user needs to know what to believe and why
• A decision requires a "state of the evidence" assessment
• deep-document-processor extracted findings from several documents

Core Methodology

Step 1: Frame the Synthesis Question

State the specific question to prevent scope creep and keep synthesis decision-relevant.

• Question: what specific claim, recommendation, or understanding should emerge?
• Decision context: what will be done with the conclusion? (Stakes determine rigor.)
• Scope boundaries: inclusion criteria — time period, domain, methodology type, minimum quality threshold

A synthesis without a clear question becomes a literature dump.

Step 2: Inventory All Evidence Sources

List every piece with provenance. Accept inputs from upstream skills:

• From causal-inference (Contract J): primary experimental data; includes quality grade, causal claim with confidence, limitations
• From learning-strategy (Contract L): varies (book, paper, expert opinion, course); per-source quality grade, key findings, gaps
• From deep-document-processor or direct input: document/dataset/report; assess quality during this step

For each source, record:

• Source identifier: name, author, date
• Source type: primary data (experimental/observational), systematic review, expert opinion, anecdote, proxy metric, model output
• Key finding: main claim or data point
• Methodology: how the finding was produced (brief)
• Relevance: how directly it addresses the synthesis question

Step 3: Grade Evidence Quality

Use the GRADE-adapted framework:

Upgrade: large effect size, dose-response, all plausible confounders would reduce the effect (conservative bias).

Downgrade: risk of bias, inconsistency across studies, indirectness (different population/setting), imprecision (wide CIs), publication bias.

A "High" grade you can't justify is really "Moderate."

Step 4: Identify Convergent Findings

Claims supported by multiple independent sources are the strongest signal.

• Group sources by claims they support
• Convergence from diverse methodologies (experiment + observation + expert agreeing) is stronger than three experiments by the same team
• Check whether convergent sources are truly independent or share data, assumptions, authors
• Weight by source quality — three High-grade converging beats five Very Low agreeing

Record: finding, supporting sources, combined quality, confidence.

Step 5: Analyze Divergent Findings

Diagnose disagreement before choosing sides:

• Methodology differences: different designs legitimately produce different results (survey vs. experiment measure different things)
• Context differences: enterprise software finding may not apply to consumer apps; 2019 result may not hold in 2026
• Scope differences: sources may measure different aspects of the same phenomenon — not actually contradicting
• Genuine disagreement: after accounting for the above, sources still disagree. Most informative — the question is harder than it looks, or one side has an unidentified flaw.

For each divergence: state the conflicting claims, diagnose likely cause, assess which to weight more (and why), flag remaining uncertainty.

Step 6: Identify Evidence Gaps

What evidence is missing that would matter?

• Questions the synthesis needs answered but no source addresses
• Quality gaps — only Very Low or Low evidence on the topic
• Perspective gaps — all sources share methodology, geography, or theoretical lens
• Recency gaps — all evidence predates a significant context change

For each gap: what evidence would fill it, importance, feasibility (feeds back to experimental-design or learning-strategy).

Step 7: Produce the Synthesis Conclusion

Integrate into a confidence-weighted conclusion:

• Primary conclusion: plain language; lead with what's most supported
• Confidence level: High (multiple High-grade converge, no significant divergence) / Moderate (some convergence with caveats) / Low (limited or conflicting) / Very Low (mostly opinion or indirect)
• Key caveats: conditions where the conclusion might not hold
• What would change the conclusion: pre-commit to revision triggers (synthesis equivalent of epistemic-mapping's mind-change conditions)

The conclusion should be decision-ready — downstream skills like decision-synthesis can take it as a criterion with the confidence level indicating weight.

Output Format

🔍 Synthesis Question

• Question: [What does the evidence say about X?]
• Decision context: [What will be decided based on this synthesis?]
• Scope: [Inclusion criteria — time, domain, methodology, quality floor]

📋 Evidence Inventory

✅ Convergent Findings

• Finding 1: [Claim supported by multiple sources]
  • Supported by: [Source #s], combined quality: [grade]
  • Independence: [Are sources truly independent?]

⚠️ Divergent Findings

• Disagreement 1: [Source A says X, Source B says Y]
  • Likely cause: [Methodology/Context/Scope/Genuine]
  • Resolution: [Which to weight more, why]
  • Remaining uncertainty: [What we still can't resolve]

🕳️ Evidence Gaps

• [Gap 1]: [What's missing, why it matters, how to fill it]

🏆 Synthesis Conclusion

• Finding: [Plain-language conclusion]
• Confidence: [High/Moderate/Low/Very Low]
• Key caveats: [Conditions where this might not hold]
• Would change conclusion if: [Pre-committed revision triggers]

Thinking Triggers

• "Am I weighting this source because it's good, or because it agrees with what I already believe?"
• "If I removed the weakest source, would the conclusion change? If so, it's resting on thin evidence."
• "Are these sources truly independent, or do they share data, methods, or authors?"
• "What evidence, if it existed, would make me conclude the opposite?"
• "Am I treating absence of evidence as evidence of absence?"
• "Would a critic point to a source I'm underweighting?"

Common Traps

• Vote counting: "3 say yes, 1 says no, so yes." Quality and effect size matter more than count. One rigorous experiment outweighs five poorly designed surveys.
• Confirmation weighting: Grading sources that support your prior as higher quality. Grade by methodology, not by agreement with your hypothesis.
• Narrative smoothing: Presenting synthesis as more coherent than evidence warrants. Real evidence is messy — acknowledge it.
• Anchoring on the first source: First evidence read disproportionately shapes the conclusion. Read challenging counter-evidence early.
• Ignoring base rates: A surprising finding from one source doesn't override strong prior evidence. Update proportionally.

Example Applications