lev-os Incremental Belief Updating
What: A forecasting and reasoning technique where you continuously update your probability estimates by small increments as new information arrives, rather than sticking to initial beliefs or making large sudden jumps.
When to use: When making predictions or estimates in uncertain domains where new evidence arrives over time—project timelines, market forecasts, risk assessments, or any judgment requiring ongoing calibration.
Introduced by: Philip Tetlock and colleagues through the Good Judgment Project (2011-2015), studying superforecasters
Core Mechanism
Bayesian-inspired updating: Each piece of new evidence should shift your probability estimate proportionally to its strength and relevance. Small evidence → small update. Large evidence → large update. No evidence → no update.
Key contrast:
- Anchoring: Stick to initial estimate despite new data
- Overcorrection: Swing wildly with each new headline
- Incremental updating: Adjust systematically in proportion to evidence weight
Why it works: Reality reveals itself gradually. Superforecasters who update frequently (but not dramatically) calibrate toward accuracy faster than those who anchor or overreact.
When to Apply
Use incremental belief updating when:
- Making predictions in domains with evolving information
- Estimating project timelines, budgets, or resource needs
- Assessing risks or probabilities over time
- Evaluating market conditions, competitive moves, or strategic options
- Tracking leading indicators for future outcomes
High-value contexts:
- Forecasting tournaments or prediction markets
- Roadmap planning and deadline estimation
- Risk management and threat assessment
- Investment decisions with ongoing information flow
- A/B test analysis as data accumulates
Execution Steps
1. Set Initial Baseline Estimate
Start with a base rate or reference class forecast. What's the historical frequency of similar outcomes? This is your prior.
2. Express Uncertainty as Probability
Instead of binary (will/won't), use probabilities: "60% chance of shipping by Q2." Enables precise updates.
3. Identify Information Triggers
What evidence would update your belief? Define upfront: "If X happens, I'll adjust by ~10% toward Y."
4. Update Small and Often
When new evidence arrives, adjust your estimate proportionally. Tetlock's superforecasters updated every few days, not just after major news.
5. Track Direction and Magnitude
Document: What changed? How strong is the evidence? By how much should I update? Did I update too much or too little?
6. Avoid Round Numbers
Don't snap to 50%, 75%, 90%. Precise estimates (63%, 78%) reflect granular thinking and force you to process evidence carefully.
7. Review Calibration Over Time
After outcomes resolve, compare your probability trajectory to reality. Were you too slow to update? Too fast? This trains judgment.
Real-World Applications
Good Judgment Project: Superforecasters who updated their predictions frequently (average 5+ times per question) and incrementally outperformed intelligence analysts with classified data.
Project Delivery Estimates: Instead of "We'll ship in 6 weeks" (then missing deadline), track probabilities: Start at 40% on-time → Sprint 1 blockers emerge: 30% → Workaround found: 45% → Scope reduced: 65%.
Market Analysis: Rather than "The market will crash" or "Bull run forever," track: "Recession probability: 25% → Fed minutes hawkish: 32% → Unemployment data weak: 28% → Credit spreads widen: 35%."
Incident Severity Assessment: Initial alert: 60% chance this is customer-impacting → More users reporting issues: 80% → Root cause identified in non-critical service: 40% → Workaround deployed: 15%.
Key Indicators
Signs you're doing it well:
- Probability estimates drift gradually, not jump wildly
- You can articulate what evidence would change your mind
- Updates correlate with information quality, not just recency
- Calibration improves over time (Brier scores decrease)
Red flags:
- Estimates haven't moved in weeks despite new information
- Large swings based on headlines without substance analysis
- Can't explain why you're at 70% vs. 65%
- Round numbers (10%, 50%, 90%) dominate estimates
Common Mistakes
Under-updating (anchoring): Sticking too close to initial estimate despite contradictory evidence. Tetlock found this more common than over-updating.
Over-updating (recency bias): Swinging dramatically with latest news without weighting historical base rates.
Binary thinking: "It will happen" or "It won't" leaves no room for calibration. Use probabilities.
Ignoring base rates: Starting from intuition rather than reference class frequencies leads to miscalibration.
Related Frameworks
Complementary: Bayesian Reasoning (formal mathematical foundation), Superforecasting (broader methodology), Reference Class Forecasting (establishing priors)
Contrasting: Gut Feel (no systematic updating), Prediction Markets (crowd-based), Scenario Planning (qualitative branches)
Sequential: Establish base rate → Set initial probability → Monitor information sources → Update proportionally → Track calibration → Adjust update strategy
Practical Examples
Feature Launch Timeline: Week 1: "70% chance we launch by end of quarter" (based on past launches) Week 3: Backend integration more complex than expected → 55% Week 5: Frontend team ahead of schedule → 60% Week 7: QA finds critical bug → 40% Week 9: Bug fixed, testing clean → 70% Result: Launched 2 days into next quarter. Probabilities tracked reality better than binary prediction.
Hiring Candidate Assessment: After resume: 40% chance of hire (base rate for this role) After phone screen: 60% (clear communication, relevant experience) After technical interview: 50% (strong on algorithms, weak on system design) After team fit interview: 70% (excellent culture match, team excited) After reference checks: 65% (one lukewarm reference, others strong) Decision: Make offer (hired, worked out well).
Security Threat Level: Start: 5% chance of attack this month (historical rate) Unusual traffic pattern detected: 8% Pattern matches known bot signature: 12% Geo-source is known threat actor region: 25% Rate limiter successfully blocking: 10% Final assessment: Elevated but contained risk, maintain monitoring.
Measurement
Quantitative signals:
- Brier Score (measures calibration, lower is better)
- Update frequency (superforecasters average 5-10 updates per question)
- Update magnitude distribution (should be mostly small, few large)
- Correlation between evidence strength and update size
Qualitative indicators:
- Can articulate reasoning for each update
- Written logs of what triggered updates
- Diversity of information sources considered
- Willingness to update in both directions (up and down)
Scoring Criteria
Practitioner Weight: 10/10 — Tetlock's research directly studied thousands of real forecasters making real predictions with measurable accuracy; superforecasters validated the approach
Clarity & Executability: 8/10 — Conceptually clear, but requires discipline and judgment about update magnitudes; easier with practice
Proven ROI: 9/10 — Good Judgment Project demonstrated dramatically improved forecasting accuracy; widely adopted in intelligence community, forecasting platforms
Novelty: 8/10 — Validates Bayesian reasoning in practical form; counterintuitive that frequent small updates beat "wait for clarity" approach
Cross-Domain Applicability: 9/10 — Applies to any domain with uncertainty and evolving information: business, engineering, investing, research, personal decisions
Total Score: 44/50 (Tier 1: Canonical)