lyndonkl Scout Mindset & Bias Check
Table of Contents
Core Principle: Map the territory accurately rather than defending a position. Forecasting requires intellectual honesty -- biases systematically distort probabilities, emotional attachment clouds judgment, and motivated reasoning leads to overconfidence.
Interactive Menu
What would you like to do?
Core Workflows
1. Run the Reversal Test - Check if you'd accept opposite evidence
- Detect motivated reasoning
- Validate evidence standards
- Expose special pleading
2. Check Scope Sensitivity - Ensure probabilities scale with inputs
- Linear scaling test
- Reference point calibration
- Magnitude assessment
3. Test Status Quo Bias - Challenge "no change" assumptions
- Entropy principle
- Change vs stability energy
- Default state inversion
4. Audit Confidence Intervals - Validate CI width
- Surprise test
- Historical calibration
- Overconfidence check
5. Run Full Bias Audit - Comprehensive bias scan
- All major cognitive biases
- Systematic checklist
- Prioritized remediation
6. Learn the Framework - Deep dive into methodology
- Read Scout vs Soldier Mindset
- Read Cognitive Bias Catalog
- Read Debiasing Techniques
7. Exit - Return to main forecasting workflow
1. Run the Reversal Test
Check if you'd accept evidence pointing the opposite direction.
Reversal Test Progress:
- [ ] Step 1: State your current conclusion
- [ ] Step 2: Identify supporting evidence
- [ ] Step 3: Reverse the evidence
- [ ] Step 4: Ask "Would I still accept it?"
- [ ] Step 5: Adjust for double standards
Step 1: State your current conclusion
What are you predicting?
- Prediction: [Event]
- Probability: [X]%
- Direction: [High/Low confidence]
Step 2: Identify supporting evidence
List the evidence that supports your conclusion.
Example: Candidate A will win (75%)
- Polls show A ahead by 5%
- A has more campaign funding
- Expert pundits favor A
- A has better debate ratings
Step 3: Reverse the evidence
Imagine the same evidence pointed the OTHER way.
Reversed: What if polls showed B ahead, B had more funding, experts favored B, and B had better ratings?
Step 4: Ask "Would I still accept it?"
The Critical Question:
If this reversed evidence existed, would I accept it as valid and change my prediction?
Three possible answers:
A) YES - I would accept reversed evidence ✓ No bias detected, continue with current reasoning
B) NO - I would dismiss reversed evidence ⚠ Warning: Motivated reasoning - you're accepting evidence when it supports you, dismissing equivalent evidence when it doesn't (special pleading)
C) UNSURE - I'd need to think about it ⚠ Warning: Asymmetric evidence standards suggest rationalizing, not reasoning
Step 5: Adjust for double standards
If you answered B or C:
Ask: Why do I dismiss this evidence in one direction but accept it in the other? Is there an objective reason, or am I motivated by preference?
Common rationalizations:
- "This source is biased" (only when it disagrees)
- "Sample size too small" (only for unfavorable polls)
- "Outlier data" (only for data you dislike)
- "Context matters" (invoked selectively)
The Fix:
- Option 1: Reject the evidence entirely (if you wouldn't trust it reversed, don't trust it now)
- Option 2: Accept it in both directions (trust evidence regardless of direction)
- Option 3: Weight it appropriately (maybe it's weak evidence both ways)
Probability adjustment: If you detected double standards, move probability 10-15% toward 50%
Next: Return to menu
2. Check Scope Sensitivity
Ensure your probabilities scale appropriately with magnitude.
Scope Sensitivity Progress:
- [ ] Step 1: Identify the variable scale
- [ ] Step 2: Test linear scaling
- [ ] Step 3: Check reference point calibration
- [ ] Step 4: Validate magnitude assessment
- [ ] Step 5: Adjust for scope insensitivity
Step 1: Identify the variable scale
What dimension has magnitude?
- Number of people (100 vs 10,000 vs 1,000,000)
- Dollar amounts ($1K vs $100K vs $10M)
- Time duration (1 month vs 1 year vs 10 years)
Step 2: Test linear scaling
The Linearity Test: Double the input, check if impact doubles.
Example: Startup funding
- If raised $1M: ___%
- If raised $10M: ___%
- If raised $100M: ___%
Scope sensitivity check: Did probabilities scale reasonably? If they barely changed → Scope insensitive
Step 3: Check reference point calibration
The Anchoring Test: Did you start with a number (base rate, someone else's forecast, round number) and insufficiently adjust?
The fix:
- Generate probability from scratch without looking at others
- Then compare and reconcile differences
- Don't just "split the difference" - reason about why estimates differ
Step 4: Validate magnitude assessment
The "1 vs 10 vs 100" Test: For your forecast, vary the scale by 10×.
Example: Project timeline
- 1 month: P(success) = ___%
- 10 months: P(success) = ___%
- 100 months: P(success) = ___%
Expected: Probability should change significantly. If all three estimates are within 10 percentage points → Scope insensitivity
Step 5: Adjust for scope insensitivity
The problem: Your emotional system responds to the category, not the magnitude.
The fix:
Method 1: Logarithmic scaling - Use log scale for intuition
Method 2: Reference class by scale - Don't use "startups" as reference class. Use "Startups that raised $1M" (10% success) vs "Startups that raised $100M" (60% success)
Method 3: Explicit calibration - Use a formula: P(success) = base_rate + k × log(amount)
Next: Return to menu
3. Test Status Quo Bias
Challenge the assumption that "no change" is the default.
Status Quo Bias Progress:
- [ ] Step 1: Identify status quo prediction
- [ ] Step 2: Calculate energy to maintain status quo
- [ ] Step 3: Invert the default
- [ ] Step 4: Apply entropy principle
- [ ] Step 5: Adjust probabilities
Step 1: Identify status quo prediction
Are you predicting "no change"? Examples: "This trend will continue," "Market share will stay the same," "Policy won't change"
Status quo predictions often get inflated probabilities because change feels risky.
Step 2: Calculate energy to maintain status quo
The Entropy Principle: In the absence of active energy input, systems decay toward disorder.
Question: "What effort is required to keep things the same?"
Examples:
- Market share: To maintain requires matching competitor innovation → Energy required: High → Status quo is HARD
- Policy: To maintain requires no proposals for change → Energy required: Low → Status quo is easier
Step 3: Invert the default
Mental Exercise:
- Normal framing: "Will X change?" (Default = no)
- Inverted framing: "Will X stay the same?" (Default = no)
Bias check: If P(change) + P(same) ≠ 100%, you have status quo bias.
Step 4: Apply entropy principle
Second Law of Thermodynamics (applied to forecasting):
Ask:
- Is this system open or closed?
- Is energy being input to maintain/improve?
- Is that energy sufficient?
Step 5: Adjust probabilities
If you detected status quo bias:
For "no change" predictions that require high energy:
- Reduce P(status quo) by 10-20%
- Increase P(change) correspondingly
For predictions where inertia truly helps: No adjustment needed
The heuristic: If maintaining status quo requires active effort, decay is more likely than you think.
Next: Return to menu
4. Audit Confidence Intervals
Validate that your CI width reflects true uncertainty.
Confidence Interval Audit Progress:
- [ ] Step 1: State current CI
- [ ] Step 2: Run surprise test
- [ ] Step 3: Check historical calibration
- [ ] Step 4: Compare to reference class variance
- [ ] Step 5: Adjust CI width
Step 1: State current CI
Current confidence interval:
- Point estimate: ___%
- Lower bound: ___%
- Upper bound: ___%
- Width: ___ percentage points
- Confidence level: ___ (usually 80% or 90%)
Step 2: Run surprise test
The Surprise Test: "Would I be genuinely shocked if the true value fell outside my confidence interval?"
Calibration:
- 80% CI → Should be shocked 20% of the time
- 90% CI → Should be shocked 10% of the time
Test: Imagine the outcome lands just below your lower bound or just above your upper bound.
Three possible answers:
- A) "Yes, I'd be very surprised" - ✓ CI appropriately calibrated
- B) "No, not that surprised" - ⚠ CI too narrow (overconfident) → Widen interval
- C) "I'd be amazed if it landed in the range" - ⚠ CI too wide → Narrow interval
Step 3: Check historical calibration
Look at your past forecasts:
- Collect last 20-50 forecasts with CIs
- Count how many actual outcomes fell outside your CIs
- Compare to theoretical expectation
| CI Level | Expected Outside | Your Actual |
|---|---|---|
| 80% | 20% | ___% |
| 90% | 10% | ___% |
Diagnosis: Actual > Expected → CIs too narrow (overconfident) - Most common
Step 4: Compare to reference class variance
If you have reference class data:
- Calculate standard deviation of reference class outcomes
- Your CI should roughly match that variance
Example: Reference class SD = 12%, your 80% CI ≈ Point estimate ± 15%
If your CI is narrower than reference class variance, you're claiming to know more than average. Justify why, or widen CI.
Step 5: Adjust CI width
Adjustment rules:
- If overconfident: Multiply current width by 1.5× to 2×
- If underconfident: Reduce width by 0.5× to 0.75×
Next: Return to menu
5. Run Full Bias Audit
Comprehensive scan of major cognitive biases.
Full Bias Audit Progress:
- [ ] Step 1: Confirmation bias check
- [ ] Step 2: Availability bias check
- [ ] Step 3: Anchoring bias check
- [ ] Step 4: Affect heuristic check
- [ ] Step 5: Overconfidence check
- [ ] Step 6: Attribution error check
- [ ] Step 7: Prioritize and remediate
See Cognitive Bias Catalog for detailed descriptions.
Quick audit questions:
1. Confirmation Bias
- Did I seek out disconfirming evidence?
- Did I give equal weight to evidence against my position?
- Did I actively try to prove myself wrong?
If NO to any → Confirmation bias detected
2. Availability Bias
- Did I rely on recent/memorable examples?
- Did I use systematic data vs "what comes to mind"?
- Did I check if my examples are representative?
If NO to any → Availability bias detected
3. Anchoring Bias
- Did I generate my estimate independently first?
- Did I avoid being influenced by others' numbers?
- Did I adjust sufficiently from initial anchor?
If NO to any → Anchoring bias detected
4. Affect Heuristic
- Do I have an emotional preference for the outcome?
- Did I separate "what I want" from "what will happen"?
- Would I make the same forecast if incentives were reversed?
If NO to any → Affect heuristic detected
5. Overconfidence
- Did I run a premortem?
- Are my CIs wide enough (surprise test)?
- Did I identify ways I could be wrong?
If NO to any → Overconfidence detected
6. Fundamental Attribution Error
- Did I attribute success to skill vs luck appropriately?
- Did I consider situational factors, not just personal traits?
- Did I avoid "great man" narratives?
If NO to any → Attribution error detected
Step 7: Prioritize and remediate
For each detected bias:
- Severity: High / Medium / Low
- Direction: Pushing probability up or down?
- Magnitude: Estimated percentage point impact
Remediation example:
| Bias | Severity | Direction | Adjustment |
|---|---|---|---|
| Confirmation | High | Up | -15% |
| Availability | Medium | Up | -10% |
| Affect heuristic | High | Up | -20% |
Net adjustment: -45% → Move probability down by 45 points (e.g., 80% → 35%)
Next: Return to menu
6. Learn the Framework
Deep dive into the methodology.
Resource Files
📄 Scout vs Soldier Mindset
- Julia Galef's framework
- Motivated reasoning
- Intellectual honesty
- Identity and beliefs
📄 Cognitive Bias Catalog
- 20+ major biases
- How they affect forecasting
- Detection methods
- Remediation strategies
📄 Debiasing Techniques
- Systematic debiasing process
- Pre-commitment strategies
- External accountability
- Algorithmic aids
Next: Return to menu
Quick Reference
The Scout Commandments
- Truth over comfort - Accuracy beats wishful thinking
- Seek disconfirmation - Try to prove yourself wrong
- Hold beliefs lightly - Probabilistic, not binary
- Update incrementally - Change mind with evidence
- Separate wanting from expecting - Desire ≠ Forecast
- Check your work - Run bias audits routinely
- Stay calibrated - Track accuracy over time
Scout mindset is the drive to see things as they are, not as you wish them to be.
Resource Files
📁 resources/
- scout-vs-soldier.md - Mindset framework
- cognitive-bias-catalog.md - Comprehensive bias reference
- debiasing-techniques.md - Remediation strategies
Ready to start? Choose a number from the menu above.