Predictor Hand Skill

Forecasting Expert Knowledge

Superforecasting Principles

Based on research by Philip Tetlock and the Good Judgment Project:

1. Triage: Focus on questions that are hard enough to be interesting but not so hard they're unknowable
2. Break problems apart: Decompose big questions into smaller, researchable sub-questions (Fermi estimation)
3. Balance inside and outside views: Use both specific evidence AND base rates from reference classes
4. Update incrementally: Adjust predictions in small steps as new evidence arrives (Bayesian updating)
5. Look for clashing forces: Identify factors pulling in opposite directions
6. Distinguish signal from noise: Weight signals by their reliability and relevance
7. Calibrate: Your 70% predictions should come true ~70% of the time
8. Post-mortem: Analyze why predictions went wrong, not just celebrate the right ones
9. Avoid the narrative trap: A compelling story is not the same as a likely outcome
10. Collaborate: Aggregate views from diverse perspectives

Signal Taxonomy

Signal Types

Signal Strength Assessment

STRONG signal (high predictive value):
  - Multiple independent sources confirm
  - Quantitative data (not just opinions)
  - Leading indicator with historical track record
  - Structural change with clear causal mechanism

MODERATE signal (some predictive value):
  - Single authoritative source
  - Expert opinion from domain specialist
  - Historical pattern that may or may not repeat
  - Lagging indicator (confirms direction)

WEAK signal (limited predictive value):
  - Social media buzz without substance
  - Single anecdote or case study
  - Rumor or unconfirmed report
  - Opinion from non-specialist

Confidence Calibration

Probability Scale

95% — Almost certain (would bet 19:1)
90% — Very likely (would bet 9:1)
80% — Likely (would bet 4:1)
70% — Probable (would bet 7:3)
60% — Slightly more likely than not
50% — Toss-up (genuine uncertainty)
40% — Slightly less likely than not
30% — Unlikely (but plausible)
20% — Very unlikely (but possible)
10% — Extremely unlikely
5%  — Almost impossible (but not zero)

Calibration Rules

1. NEVER use 0% or 100% — nothing is absolutely certain
2. If you haven't done research, default to the base rate (outside view)
3. Your first estimate should be the reference class base rate
4. Adjust from the base rate using specific evidence (inside view)
5. Typical adjustment: ±5-15% per strong signal, ±2-5% per moderate signal
6. If your gut says 80% but your analysis says 55%, trust the analysis

Brier Score

The gold standard for measuring prediction accuracy:

Brier Score = (predicted_probability - actual_outcome)^2

actual_outcome = 1 if prediction came true, 0 if not

Perfect score: 0.0 (you're always right with perfect confidence)
Coin flip: 0.25 (saying 50% on everything)
Terrible: 1.0 (100% confident, always wrong)

Good forecaster: < 0.15
Average forecaster: 0.20-0.30
Bad forecaster: > 0.35

Domain-Specific Source Guide

Technology Predictions

Finance Predictions

Geopolitics Predictions

Climate Predictions

Reasoning Chain Construction

Template

PREDICTION: [Specific, falsifiable claim]

1. REFERENCE CLASS (Outside View)
   Base rate: [What % of similar events occur?]
   Reference examples: [3-5 historical analogues]

2. SPECIFIC EVIDENCE (Inside View)
   Signals FOR (+):
   a. [Signal] — strength: [strong/moderate/weak] — adjustment: +X%
   b. [Signal] — strength: [strong/moderate/weak] — adjustment: +X%

   Signals AGAINST (-):
   a. [Signal] — strength: [strong/moderate/weak] — adjustment: -X%
   b. [Signal] — strength: [strong/moderate/weak] — adjustment: -X%

3. SYNTHESIS
   Starting probability (base rate): X%
   Net adjustment: +/-Y%
   Final probability: Z%

4. KEY ASSUMPTIONS
   - [Assumption 1]: If wrong, probability shifts to [W%]
   - [Assumption 2]: If wrong, probability shifts to [V%]

5. RESOLUTION
   Date: [When can this be resolved?]
   Criteria: [Exactly how to determine if correct]
   Data source: [Where to check the outcome]

Prediction Tracking & Scoring

Prediction Ledger Format

{
  "id": "pred_001",
  "created": "2025-01-15",
  "prediction": "OpenAI will release GPT-5 before July 2025",
  "confidence": 0.65,
  "domain": "tech",
  "time_horizon": "2025-07-01",
  "reasoning_chain": "...",
  "key_signals": ["leaked roadmap", "compute scaling", "hiring patterns"],
  "status": "active|resolved|expired",
  "resolution": {
    "date": "2025-06-30",
    "outcome": true,
    "evidence": "Released June 15, 2025",
    "brier_score": 0.1225
  },
  "updates": [
    {"date": "2025-03-01", "new_confidence": 0.75, "reason": "New evidence: leaked demo"}
  ]
}

Accuracy Report Template

ACCURACY DASHBOARD
==================
Total predictions:     N
Resolved predictions:  N (N correct, N incorrect, N partial)
Active predictions:    N
Expired (unresolvable):N

Overall accuracy:      X%
Brier score:           0.XX

Calibration:
  Predicted 90%+ → Actual: X% (N predictions)
  Predicted 70-89% → Actual: X% (N predictions)
  Predicted 50-69% → Actual: X% (N predictions)
  Predicted 30-49% → Actual: X% (N predictions)
  Predicted <30% → Actual: X% (N predictions)

Strengths: [domains/types where you perform well]
Weaknesses: [domains/types where you perform poorly]

Cognitive Bias Checklist

Before finalizing any prediction, check for these biases:

1. Anchoring: Am I fixated on the first number I encountered?

   • Fix: Deliberately consider the base rate before looking at specific evidence

2. Availability bias: Am I overweighting recent or memorable events?

   • Fix: Check the actual frequency, not just what comes to mind

3. Confirmation bias: Am I only looking for evidence that supports my prediction?

   • Fix: Actively search for contradicting evidence (steel-man the opposite)

4. Narrative bias: Am I choosing a prediction because it makes a good story?

   • Fix: Boring predictions are often more accurate

5. Overconfidence: Am I too sure?

   • Fix: If you've never been wrong at this confidence level, you're probably overconfident

6. Scope insensitivity: Am I treating very different scales the same?

   • Fix: Be specific about magnitudes and timeframes

7. Recency bias: Am I extrapolating recent trends too far?

   • Fix: Check longer time horizons and mean reversion patterns

8. Status quo bias: Am I defaulting to "nothing will change"?

   • Fix: Consider structural changes that could break the status quo

Contrarian Mode

When enabled, for each consensus prediction:

1. Identify what the consensus view is
2. Search for evidence the consensus is wrong
3. Consider: "What would have to be true for the opposite to happen?"
4. If credible contrarian evidence exists, include a contrarian prediction
5. Always label contrarian predictions clearly with the consensus for comparison