majiayu000 Scout Mindset & Bias Check
Table of Contents
What is Scout Mindset?
Scout Mindset (Julia Galef) is the motivation to see things as they are, not as you wish them to be. Contrast with Soldier Mindset, which defends a position regardless of evidence.
Core Principle: Your goal is to map the territory accurately, not win an argument.
Why It Matters:
- Forecasting requires intellectual honesty
- Biases systemically distort probabilities
- Emotional attachment clouds judgment
- Motivated reasoning leads to overconfidence
When to Use This Skill
Use this skill when:
- Prediction feels emotional - You want a certain outcome
- Stuck at 50/50 - Indecisive, can't commit to probability
- Defending a position - Arguing for your forecast, not questioning it
- After inside view analysis - Used specific details, need bias check
- Disagreement with others - Different people, different probabilities
- Before finalizing - Last sanity check
Do NOT skip this when stakes are high, you have strong priors, or forecast affects you personally.
Interactive Menu
What would you like to do?
Core Workflows
1. Run the Reversal Test - Check if you'd accept opposite evidence
- Detect motivated reasoning
- Validate evidence standards
- Expose special pleading
2. Check Scope Sensitivity - Ensure probabilities scale with inputs
- Linear scaling test
- Reference point calibration
- Magnitude assessment
3. Test Status Quo Bias - Challenge "no change" assumptions
- Entropy principle
- Change vs stability energy
- Default state inversion
4. Audit Confidence Intervals - Validate CI width
- Surprise test
- Historical calibration
- Overconfidence check
5. Run Full Bias Audit - Comprehensive bias scan
- All major cognitive biases
- Systematic checklist
- Prioritized remediation
6. Learn the Framework - Deep dive into methodology
- Read Scout vs Soldier Mindset
- Read Cognitive Bias Catalog
- Read Debiasing Techniques
7. Exit - Return to main forecasting workflow
1. Run the Reversal Test
Check if you'd accept evidence pointing the opposite direction.
Reversal Test Progress:
- [ ] Step 1: State your current conclusion
- [ ] Step 2: Identify supporting evidence
- [ ] Step 3: Reverse the evidence
- [ ] Step 4: Ask "Would I still accept it?"
- [ ] Step 5: Adjust for double standards
Step 1: State your current conclusion
What are you predicting?
- Prediction: [Event]
- Probability: [X]%
- Direction: [High/Low confidence]
Step 2: Identify supporting evidence
List the evidence that supports your conclusion.
Example: Candidate A will win (75%)
- Polls show A ahead by 5%
- A has more campaign funding
- Expert pundits favor A
- A has better debate ratings
Step 3: Reverse the evidence
Imagine the same evidence pointed the OTHER way.
Reversed: What if polls showed B ahead, B had more funding, experts favored B, and B had better ratings?
Step 4: Ask "Would I still accept it?"
The Critical Question:
If this reversed evidence existed, would I accept it as valid and change my prediction?
Three possible answers:
A) YES - I would accept reversed evidence ✓ No bias detected, continue with current reasoning
B) NO - I would dismiss reversed evidence ⚠ Warning: Motivated reasoning - you're accepting evidence when it supports you, dismissing equivalent evidence when it doesn't (special pleading)
C) UNSURE - I'd need to think about it ⚠ Warning: Asymmetric evidence standards suggest rationalizing, not reasoning
Step 5: Adjust for double standards
If you answered B or C:
Ask: Why do I dismiss this evidence in one direction but accept it in the other? Is there an objective reason, or am I motivated by preference?
Common rationalizations:
- "This source is biased" (only when it disagrees)
- "Sample size too small" (only for unfavorable polls)
- "Outlier data" (only for data you dislike)
- "Context matters" (invoked selectively)
The Fix:
- Option 1: Reject the evidence entirely (if you wouldn't trust it reversed, don't trust it now)
- Option 2: Accept it in both directions (trust evidence regardless of direction)
- Option 3: Weight it appropriately (maybe it's weak evidence both ways)
Probability adjustment: If you detected double standards, move probability 10-15% toward 50%
Next: Return to menu
2. Check Scope Sensitivity
Ensure your probabilities scale appropriately with magnitude.
Scope Sensitivity Progress:
- [ ] Step 1: Identify the variable scale
- [ ] Step 2: Test linear scaling
- [ ] Step 3: Check reference point calibration
- [ ] Step 4: Validate magnitude assessment
- [ ] Step 5: Adjust for scope insensitivity
Step 1: Identify the variable scale
What dimension has magnitude?
- Number of people (100 vs 10,000 vs 1,000,000)
- Dollar amounts ($1K vs $100K vs $10M)
- Time duration (1 month vs 1 year vs 10 years)
Step 2: Test linear scaling
The Linearity Test: Double the input, check if impact doubles.
Example: Startup funding
- If raised $1M: ___%
- If raised $10M: ___%
- If raised $100M: ___%
Scope sensitivity check: Did probabilities scale reasonably? If they barely changed → Scope insensitive
Step 3: Check reference point calibration
The Anchoring Test: Did you start with a number (base rate, someone else's forecast, round number) and insufficiently adjust?
The fix:
- Generate probability from scratch without looking at others
- Then compare and reconcile differences
- Don't just "split the difference" - reason about why estimates differ
Step 4: Validate magnitude assessment
The "1 vs 10 vs 100" Test: For your forecast, vary the scale by 10×.
Example: Project timeline
- 1 month: P(success) = ___%
- 10 months: P(success) = ___%
- 100 months: P(success) = ___%
Expected: Probability should change significantly. If all three estimates are within 10 percentage points → Scope insensitivity
Step 5: Adjust for scope insensitivity
The problem: Your emotional system responds to the category, not the magnitude.
The fix:
Method 1: Logarithmic scaling - Use log scale for intuition
Method 2: Reference class by scale - Don't use "startups" as reference class. Use "Startups that raised $1M" (10% success) vs "Startups that raised $100M" (60% success)
Method 3: Explicit calibration - Use a formula: P(success) = base_rate + k × log(amount)
Next: Return to menu
3. Test Status Quo Bias
Challenge the assumption that "no change" is the default.
Status Quo Bias Progress:
- [ ] Step 1: Identify status quo prediction
- [ ] Step 2: Calculate energy to maintain status quo
- [ ] Step 3: Invert the default
- [ ] Step 4: Apply entropy principle
- [ ] Step 5: Adjust probabilities
Step 1: Identify status quo prediction
Are you predicting "no change"? Examples: "This trend will continue," "Market share will stay the same," "Policy won't change"
Status quo predictions often get inflated probabilities because change feels risky.
Step 2: Calculate energy to maintain status quo
The Entropy Principle: In the absence of active energy input, systems decay toward disorder.
Question: "What effort is required to keep things the same?"
Examples:
- Market share: To maintain requires matching competitor innovation → Energy required: High → Status quo is HARD
- Policy: To maintain requires no proposals for change → Energy required: Low → Status quo is easier
Step 3: Invert the default
Mental Exercise:
- Normal framing: "Will X change?" (Default = no)
- Inverted framing: "Will X stay the same?" (Default = no)
Bias check: If P(change) + P(same) ≠ 100%, you have status quo bias.
Step 4: Apply entropy principle
Second Law of Thermodynamics (applied to forecasting):
Ask:
- Is this system open or closed?
- Is energy being input to maintain/improve?
- Is that energy sufficient?
Step 5: Adjust probabilities
If you detected status quo bias:
For "no change" predictions that require high energy:
- Reduce P(status quo) by 10-20%
- Increase P(change) correspondingly
For predictions where inertia truly helps: No adjustment needed
The heuristic: If maintaining status quo requires active effort, decay is more likely than you think.
Next: Return to menu
4. Audit Confidence Intervals
Validate that your CI width reflects true uncertainty.
Confidence Interval Audit Progress:
- [ ] Step 1: State current CI
- [ ] Step 2: Run surprise test
- [ ] Step 3: Check historical calibration
- [ ] Step 4: Compare to reference class variance
- [ ] Step 5: Adjust CI width
Step 1: State current CI
Current confidence interval:
- Point estimate: ___%
- Lower bound: ___%
- Upper bound: ___%
- Width: ___ percentage points
- Confidence level: ___ (usually 80% or 90%)
Step 2: Run surprise test
The Surprise Test: "Would I be genuinely shocked if the true value fell outside my confidence interval?"
Calibration:
- 80% CI → Should be shocked 20% of the time
- 90% CI → Should be shocked 10% of the time
Test: Imagine the outcome lands just below your lower bound or just above your upper bound.
Three possible answers:
- A) "Yes, I'd be very surprised" - ✓ CI appropriately calibrated
- B) "No, not that surprised" - ⚠ CI too narrow (overconfident) → Widen interval
- C) "I'd be amazed if it landed in the range" - ⚠ CI too wide → Narrow interval
Step 3: Check historical calibration
Look at your past forecasts:
- Collect last 20-50 forecasts with CIs
- Count how many actual outcomes fell outside your CIs
- Compare to theoretical expectation
| CI Level | Expected Outside | Your Actual |
|---|---|---|
| 80% | 20% | ___% |
| 90% | 10% | ___% |
Diagnosis: Actual > Expected → CIs too narrow (overconfident) - Most common
Step 4: Compare to reference class variance
If you have reference class data:
- Calculate standard deviation of reference class outcomes
- Your CI should roughly match that variance
Example: Reference class SD = 12%, your 80% CI ≈ Point estimate ± 15%
If your CI is narrower than reference class variance, you're claiming to know more than average. Justify why, or widen CI.
Step 5: Adjust CI width
Adjustment rules:
- If overconfident: Multiply current width by 1.5× to 2×
- If underconfident: Reduce width by 0.5× to 0.75×
Next: Return to menu
5. Run Full Bias Audit
Comprehensive scan of major cognitive biases.
Full Bias Audit Progress:
- [ ] Step 1: Confirmation bias check
- [ ] Step 2: Availability bias check
- [ ] Step 3: Anchoring bias check
- [ ] Step 4: Affect heuristic check
- [ ] Step 5: Overconfidence check
- [ ] Step 6: Attribution error check
- [ ] Step 7: Prioritize and remediate
See Cognitive Bias Catalog for detailed descriptions.
Quick audit questions:
1. Confirmation Bias
- Did I seek out disconfirming evidence?
- Did I give equal weight to evidence against my position?
- Did I actively try to prove myself wrong?
If NO to any → Confirmation bias detected
2. Availability Bias
- Did I rely on recent/memorable examples?
- Did I use systematic data vs "what comes to mind"?
- Did I check if my examples are representative?
If NO to any → Availability bias detected
3. Anchoring Bias
- Did I generate my estimate independently first?
- Did I avoid being influenced by others' numbers?
- Did I adjust sufficiently from initial anchor?
If NO to any → Anchoring bias detected
4. Affect Heuristic
- Do I have an emotional preference for the outcome?
- Did I separate "what I want" from "what will happen"?
- Would I make the same forecast if incentives were reversed?
If NO to any → Affect heuristic detected
5. Overconfidence
- Did I run a premortem?
- Are my CIs wide enough (surprise test)?
- Did I identify ways I could be wrong?
If NO to any → Overconfidence detected
6. Fundamental Attribution Error
- Did I attribute success to skill vs luck appropriately?
- Did I consider situational factors, not just personal traits?
- Did I avoid "great man" narratives?
If NO to any → Attribution error detected
Step 7: Prioritize and remediate
For each detected bias:
- Severity: High / Medium / Low
- Direction: Pushing probability up or down?
- Magnitude: Estimated percentage point impact
Remediation example:
| Bias | Severity | Direction | Adjustment |
|---|---|---|---|
| Confirmation | High | Up | -15% |
| Availability | Medium | Up | -10% |
| Affect heuristic | High | Up | -20% |
Net adjustment: -45% → Move probability down by 45 points (e.g., 80% → 35%)
Next: Return to menu
6. Learn the Framework
Deep dive into the methodology.
Resource Files
📄 Scout vs Soldier Mindset
- Julia Galef's framework
- Motivated reasoning
- Intellectual honesty
- Identity and beliefs
📄 Cognitive Bias Catalog
- 20+ major biases
- How they affect forecasting
- Detection methods
- Remediation strategies
📄 Debiasing Techniques
- Systematic debiasing process
- Pre-commitment strategies
- External accountability
- Algorithmic aids
Next: Return to menu
Quick Reference
The Scout Commandments
- Truth over comfort - Accuracy beats wishful thinking
- Seek disconfirmation - Try to prove yourself wrong
- Hold beliefs lightly - Probabilistic, not binary
- Update incrementally - Change mind with evidence
- Separate wanting from expecting - Desire ≠ Forecast
- Check your work - Run bias audits routinely
- Stay calibrated - Track accuracy over time
Scout mindset is the drive to see things as they are, not as you wish them to be.
Resource Files
📁 resources/
- scout-vs-soldier.md - Mindset framework
- cognitive-bias-catalog.md - Comprehensive bias reference
- debiasing-techniques.md - Remediation strategies
Ready to start? Choose a number from the menu above.