tools-only Common Biases in Scientific Research
Cognitive Biases Affecting Researchers
1. Confirmation Bias
Description: Tendency to search for, interpret, and recall information that confirms preexisting beliefs.
Manifestations:
- Designing studies that can only support the hypothesis
- Interpreting ambiguous results as supportive
- Remembering hits and forgetting misses
- Selectively citing literature that agrees
Mitigation:
- Preregister hypotheses and analysis plans
- Actively seek disconfirming evidence
- Use blinded data analysis
- Consider alternative hypotheses
2. Hindsight Bias (I-Knew-It-All-Along Effect)
Description: After an event, people perceive it as having been more predictable than it actually was.
Manifestations:
- HARKing (Hypothesizing After Results are Known)
- Claiming predictions that weren't made
- Underestimating surprise at results
Mitigation:
- Document predictions before data collection
- Preregister studies
- Distinguish exploratory from confirmatory analyses
3. Publication Bias (File Drawer Problem)
Description: Positive/significant results are more likely to be published than negative/null results.
Manifestations:
- Literature appears to support effects that don't exist
- Overestimation of effect sizes
- Inability to estimate true effects from published literature
Mitigation:
- Publish null results
- Use preregistration and registered reports
- Conduct systematic reviews with grey literature
- Check for funnel plot asymmetry in meta-analyses
4. Anchoring Bias
Description: Over-reliance on the first piece of information encountered.
Manifestations:
- Initial hypotheses unduly influence interpretation
- First studies in a field set expectations
- Pilot data biases main study interpretation
Mitigation:
- Consider multiple initial hypotheses
- Evaluate evidence independently
- Use structured decision-making
5. Availability Heuristic
Description: Overestimating likelihood of events based on how easily examples come to mind.
Manifestations:
- Overemphasizing recent or dramatic findings
- Neglecting base rates
- Anecdotal evidence overshadowing statistics
Mitigation:
- Consult systematic reviews, not memorable papers
- Consider base rates explicitly
- Use statistical thinking, not intuition
6. Bandwagon Effect
Description: Adopting beliefs because many others hold them.
Manifestations:
- Following research trends without critical evaluation
- Citing widely-cited papers without reading
- Accepting "textbook knowledge" uncritically
Mitigation:
- Evaluate evidence independently
- Read original sources
- Question assumptions
7. Belief Perseverance
Description: Maintaining beliefs even after evidence disproving them.
Manifestations:
- Defending theories despite contradictory evidence
- Finding ad hoc explanations for discrepant results
- Dismissing replication failures
Mitigation:
- Explicitly consider what evidence would change your mind
- Update beliefs based on evidence
- Distinguish between theories and ego
8. Outcome Bias
Description: Judging decisions based on outcomes rather than the quality of the decision at the time.
Manifestations:
- Valuing lucky guesses over sound methodology
- Dismissing good studies with null results
- Rewarding sensational findings over rigorous methods
Mitigation:
- Evaluate methodology independently of results
- Value rigor and transparency
- Recognize role of chance
Experimental and Methodological Biases
9. Selection Bias
Description: Systematic differences between those selected for study and those not selected.
Types:
- Sampling bias: Non-random sample
- Attrition bias: Systematic dropout
- Volunteer bias: Self-selected participants differ
- Berkson's bias: Hospital patients differ from general population
- Survivorship bias: Only examining "survivors"
Detection:
- Compare characteristics of participants vs. target population
- Analyze dropout patterns
- Consider who is missing from the sample
Mitigation:
- Random sampling
- Track and analyze non-responders
- Use strategies to minimize dropout
- Report participant flow diagrams
10. Observer Bias (Detection Bias)
Description: Researchers' expectations influence observations or measurements.
Manifestations:
- Measuring outcomes differently across groups
- Interpreting ambiguous results based on group assignment
- Unconsciously cueing participants
Mitigation:
- Blinding of observers/assessors
- Objective, automated measurements
- Standardized protocols
- Inter-rater reliability checks
11. Performance Bias
Description: Systematic differences in care provided to comparison groups.
Manifestations:
- Treating experimental group differently
- Providing additional attention to one group
- Differential adherence to protocols
Mitigation:
- Standardize all procedures
- Blind participants and providers
- Use placebo controls
- Monitor protocol adherence
12. Measurement Bias (Information Bias)
Description: Systematic errors in how variables are measured.
Types:
- Recall bias: Systematic differences in accuracy of recall
- Social desirability bias: Responding in socially acceptable ways
- Interviewer bias: Interviewer's characteristics affect responses
- Instrument bias: Measurement tools systematically err
Mitigation:
- Use validated, objective measures
- Standardize data collection
- Blind participants to hypotheses
- Verify self-reports with objective data
13. Confounding Bias
Description: Effect of extraneous variable mixed with the variable of interest.
Examples:
- Age confounding relationship between exercise and health
- Socioeconomic status confounding education and outcomes
- Indication bias in treatment studies
Mitigation:
- Randomization
- Matching
- Statistical adjustment
- Stratification
- Restriction
14. Reporting Bias
Description: Selective reporting of results.
Types:
- Outcome reporting bias: Selectively reporting outcomes
- Time-lag bias: Delayed publication of negative results
- Language bias: Publishing positive results in English
- Citation bias: Preferentially citing positive studies
Mitigation:
- Preregister all outcomes
- Report all planned analyses
- Distinguish primary from secondary outcomes
- Use study registries
15. Spectrum Bias
Description: Test performance varies depending on the spectrum of disease severity in the sample.
Manifestations:
- Diagnostic tests appearing more accurate in extreme cases
- Treatment effects differing by severity
Mitigation:
- Test in representative samples
- Report performance across disease spectrum
- Avoid case-control designs for diagnostic studies
16. Lead-Time Bias
Description: Apparent survival benefit due to earlier detection, not improved outcomes.
Example:
- Screening detecting disease earlier makes survival seem longer, even if death occurs at same age
Mitigation:
- Measure mortality, not just survival from diagnosis
- Use randomized screening trials
- Consider length-time and overdiagnosis bias
17. Length-Time Bias
Description: Screening disproportionately detects slower-growing, less aggressive cases.
Example:
- Slow-growing cancers detected more often than fast-growing ones, making screening appear beneficial
Mitigation:
- Randomized trials with mortality endpoints
- Consider disease natural history
18. Response Bias
Description: Systematic pattern in how participants respond.
Types:
- Acquiescence bias: Tendency to agree
- Extreme responding: Always choosing extreme options
- Neutral responding: Avoiding extreme responses
- Demand characteristics: Responding based on perceived expectations
Mitigation:
- Mix positive and negative items
- Use multiple response formats
- Blind participants to hypotheses
- Use behavioral measures
Statistical and Analysis Biases
19. P-Hacking (Data Dredging)
Description: Manipulating data or analyses until significant results emerge.
Manifestations:
- Collecting data until significance reached
- Testing multiple outcomes, reporting only significant ones
- Trying multiple analysis methods
- Excluding "outliers" to reach significance
- Subgroup analyses until finding significance
Detection:
- Suspiciously perfect p-values (just below .05)
- Many researcher degrees of freedom
- Undisclosed analyses
- Fishing expeditions
Mitigation:
- Preregister analysis plans
- Report all analyses conducted
- Correct for multiple comparisons
- Distinguish exploratory from confirmatory
20. HARKing (Hypothesizing After Results are Known)
Description: Presenting post hoc hypotheses as if they were predicted a priori.
Why problematic:
- Inflates apparent evidence
- Conflates exploration with confirmation
- Misrepresents the scientific process
Mitigation:
- Preregister hypotheses
- Clearly label exploratory analyses
- Require replication of unexpected findings
21. Base Rate Neglect
Description: Ignoring prior probability when evaluating evidence.
Example:
- Test with 95% accuracy in rare disease (1% prevalence): positive result only 16% likely to indicate disease
Mitigation:
- Always consider base rates/prior probability
- Use Bayesian reasoning
- Report positive and negative predictive values
22. Regression to the Mean
Description: Extreme measurements tend to be followed by less extreme ones.
Manifestations:
- Treatment effects in extreme groups may be regression artifacts
- "Sophomore slump" in high performers
Mitigation:
- Use control groups
- Consider natural variation
- Don't select based on extreme baseline values without controls
23. Texas Sharpshooter Fallacy
Description: Selecting data after seeing patterns, like shooting arrows then drawing targets around clusters.
Manifestations:
- Finding patterns in random data
- Subgroup analyses selected post hoc
- Geographic clustering studies without correction
Mitigation:
- Prespecify hypotheses
- Correct for multiple comparisons
- Replicate findings in independent data
Reducing Bias: Best Practices
Study Design
- Randomization
- Blinding (single, double, triple)
- Control groups
- Adequate sample size
- Preregistration
Data Collection
- Standardized protocols
- Validated instruments
- Objective measures when possible
- Multiple observers/raters
- Complete data collection
Analysis
- Intention-to-treat analysis
- Prespecified analyses
- Appropriate statistical tests
- Multiple comparison corrections
- Sensitivity analyses
Reporting
- Complete transparency
- CONSORT, PRISMA, or similar guidelines
- Report all outcomes
- Distinguish exploratory from confirmatory
- Share data and code
Meta-Level
- Adversarial collaboration
- Replication studies
- Open science practices
- Peer review
- Systematic reviews