Superforecasting

Superforecasting

Category: Decision-Making & Strategic Thinking Source: Philip Tetlock & Dan Gardner - "Superforecasting: The Art and Science of Prediction" (2015) Practitioner Score: 46/50 (Tier 1 Canonical)

Overview

Superforecasting is a systematic methodology for making accurate probabilistic predictions, developed through Philip Tetlock's Good Judgment Project. The research identified "superforecasters" - individuals who consistently outperform experts, pundits, and even intelligence analysts by 30% or more. The framework codifies their techniques into 10 actionable commandments plus deliberate practice protocols.

Core Insight: Prediction accuracy is a learnable skill. By combining rigorous process (break problems down, update beliefs incrementally, balance outside/inside views) with calibrated probabilistic thinking and error analysis, anyone can dramatically improve forecasting ability.

Evidence: Good Judgment Project participants trained in these techniques beat CIA analysts with classified information access.

When to Use

• Strategic decisions - Market entry, product launches, competitive moves
• Resource allocation - Investment decisions, hiring plans, capacity planning
• Risk assessment - Project timelines, crisis likelihood, threat analysis
• Competitive intelligence - Predicting competitor actions, market shifts
• Long-range planning - Technology adoption curves, regulatory changes

Anti-patterns:

• Decisions already made (confirmation bias trap)
• Binary yes/no thinking without probability ranges
• One-shot predictions without learning feedback
• High-emotion situations without cooling-off period

How to Execute

Step 1: Triage - Choose Forecast-Worthy Questions

Action: Focus effort on questions where hard work pays off

• Skip "clocklike" questions: Simple rules/trends suffice (e.g., "Will sun rise tomorrow?")
• Skip "cloud-like" questions: Too random even for models (e.g., "Will World War III start?")
• Target Goldilocks zone: Difficult but tractable with analysis
• Output: Prioritized list of forecastable questions

Step 2: Fermi-Style Decomposition

Action: Break intractable problems into tractable sub-problems

• Identify knowable parts: What can be researched or estimated?
• Expose unknowables: Flush ignorance into the open
• Example: "Will product X succeed?" → market size × conversion rate × pricing × competition
• Output: Hierarchical breakdown with researchable components

Step 3: Balance Outside View (Base Rates) and Inside View

Action: Start with reference class, adjust for specifics

• Outside view first: How often do things of this sort happen in situations of this sort?
• Find comparison class: Similar products, markets, technologies
• Inside view adjustment: What makes this case unique?
• Output: Base rate probability + reasoned adjustments

Step 4: Incremental Belief Updating

Action: Update forecasts as new evidence arrives - not too much, not too little

• Avoid under-reacting: Ignoring genuinely new information
• Avoid over-reacting: Jumping to conclusions from noisy signals
• Bayesian mindset: P(H|E) = P(E|H) × P(H) / P(E)
• Output: Revised probability with explicit reasoning

Step 5: Consider Clashing Causal Forces

Action: Map arguments for AND against your thesis

• Steel-man opposition: Understand counterarguments deeply
• Force interaction: How do conflicting factors balance?
• Example: "AI adoption" → Cost savings (pro) vs. Implementation complexity (con)
• Output: Two-column list of forces with relative weights

Step 6: Granular Probability Estimates

Action: Translate vague hunches into numeric probabilities

• Avoid vague language: "Likely" means what exactly?
• Use fine gradations: 55% vs. 60% forces precision
• Calibration practice: Track how often your 70% predictions come true
• Output: Numeric probability (e.g., 68%) with confidence range

Step 7: Balance Under/Overconfidence

Action: Manage trade-off between decisiveness and humility

• Calibration: Are your 80% predictions correct 80% of the time?
• Resolution: Can you distinguish 60% from 80% events?
• Avoid extremes: "Definitely" (99%+) and "No way" (1%-) rarely justified
• Output: Calibrated probability that neither overstates nor understates certainty

Step 8: Learn from Errors Without Hindsight Bias

Action: Analyze mistakes while resisting "I knew it all along"

• Pre-mortem: Before outcome, write why forecast might fail
• Post-mortem: After outcome, compare to pre-mortem (not current knowledge)
• Brier score tracking: Measure accuracy over time
• Output: Error log with root cause analysis

Step 9: Leverage Team Wisdom

Action: Master collaborative forecasting dynamics

• Perspective-taking: Reproduce others' arguments to their satisfaction
• Precision questioning: Help clarify without judgment
• Constructive confrontation: Disagree without being disagreeable
• Output: Team forecast incorporating diverse viewpoints

Step 10: Master the Error-Balancing Bicycle

Action: Treat commandments as guidelines requiring constant judgment

• No rigid rules: Every situation is unique
• Deliberate practice: Forecasting is skill built through repetition
• Feedback loops: Clear, unambiguous results inform learning
• Output: Continuous improvement trajectory

Real-World Examples

Good Judgment Project (2011-2015):

• Superforecasters beat intelligence analysts by 30%
• Ordinary people trained in these methods outperformed experts
• Result: Validated that forecasting is a learnable skill

Prediction Markets (Metaculus, Good Judgment Open):

• Calibrated forecasters consistently identify probability ranges
• Aggregated predictions outperform individual experts
• Result: Operational use in policy, business, research

Tech Industry Product Forecasting:

• Decompose adoption rates into addressable market × conversion × retention
• Update predictions as beta data arrives
• Result: Better resource allocation, realistic roadmaps

Integration Points

Complements:

• Brier Score: Measures superforecasting accuracy quantitatively
• Fermi Estimation: Powers Step 2 decomposition
• Bayes' Theorem: Mathematical foundation for belief updating
• Calibration: Essential skill for Step 7 confidence management
• Base Rate Analysis: Core of Step 3 outside view

Contrasts with:

• Expert Intuition: Systematic process vs. gut feel
• Punditry: Probabilistic humility vs. confident pronouncements
• Binary Thinking: 65% vs. "yes/no"

Common Pitfalls

Pitfall 1: Anchoring on Initial Estimate

• Warning sign: Forecast barely moves despite major news
• Fix: Explicit belief updating protocol after each information update

Pitfall 2: Ignoring Base Rates

• Warning sign: "This time is different" without evidence
• Fix: Always start with outside view reference class

Pitfall 3: Overconfidence in Extremes

• Warning sign: Many forecasts at 5% or 95%
• Fix: Force justification for extreme probabilities, track calibration

Pitfall 4: Confirmation Bias in Research

• Warning sign: Only seeking evidence supporting initial view
• Fix: Actively search for disconfirming evidence (Step 5)

Pitfall 5: No Feedback Loop

• Warning sign: Making predictions but never tracking outcomes
• Fix: Maintain prediction log with dates, probabilities, and resolutions

Validation Checklist

• Question is in Goldilocks zone (neither trivial nor impossible)
• Problem decomposed into researchable sub-components
• Base rate identified from reference class
• Both supporting and opposing forces mapped
• Probability is numeric and granular (not vague language)
• Calibration tracked over time (70% predictions = 70% accuracy)
• Forecast updated as new information arrives
• Pre-mortem written before outcome known
• Team input incorporated through structured dialogue

Key Metrics

Brier Score: Primary accuracy measure (0 = perfect, 2 = worst)

• Formula: (1/N) Σ(forecast - outcome)²
• Target: < 0.20 for well-calibrated forecaster

Calibration: Do your X% predictions happen X% of the time?

• Plot predicted probability vs. observed frequency
• Perfect calibration = diagonal line

Resolution: Can you distinguish different probability levels?

• Difference in outcomes between 60% and 80% forecasts
• Higher resolution = better discrimination

Further Reading

• "Superforecasting" - Philip Tetlock & Dan Gardner (2015)
• "Expert Political Judgment" - Philip Tetlock (2005)
• Good Judgment Open: Free forecasting platform with training
• Metaculus: Advanced forecasting community
• "The Signal and the Noise" - Nate Silver (Bayesian thinking)