tools-only Scientific Method Core Principles
Fundamental Principles
1. Empiricism
- Knowledge derives from observable, measurable evidence
- Claims must be testable through observation or experiment
- Subjective experience alone is insufficient for scientific conclusions
2. Falsifiability (Popper's Criterion)
- A hypothesis must be capable of being proven false
- Unfalsifiable claims are not scientific (e.g., "invisible, undetectable forces")
- Good hypotheses make specific, testable predictions
3. Reproducibility
- Results must be replicable by independent researchers
- Methods must be described with sufficient detail for replication
- Single studies are rarely definitive; replication strengthens confidence
4. Parsimony (Occam's Razor)
- Prefer simpler explanations over complex ones when both fit the data
- Don't multiply entities unnecessarily
- Extraordinary claims require extraordinary evidence
5. Systematic Observation
- Use standardized, rigorous methods
- Control for confounding variables
- Minimize observer bias through blinding and protocols
The Scientific Process
1. Question Formation
- Identify a specific, answerable question
- Ensure the question is within the scope of scientific inquiry
- Consider whether current methods can address the question
2. Literature Review
- Survey existing knowledge
- Identify gaps and contradictions
- Build on previous work rather than reinventing
3. Hypothesis Development
- State a clear, testable prediction
- Define variables operationally
- Specify the expected relationship between variables
4. Experimental Design
- Choose appropriate methodology
- Identify independent and dependent variables
- Control confounding variables
- Select appropriate sample size and population
- Plan statistical analyses in advance
5. Data Collection
- Follow protocols consistently
- Record all observations, including unexpected results
- Maintain detailed lab notebooks or data logs
- Use validated measurement instruments
6. Analysis
- Apply appropriate statistical methods
- Test assumptions of statistical tests
- Consider effect size, not just significance
- Look for alternative explanations
7. Interpretation
- Distinguish between correlation and causation
- Acknowledge limitations
- Consider alternative interpretations
- Avoid overgeneralizing beyond the data
8. Communication
- Report methods transparently
- Include negative results
- Acknowledge conflicts of interest
- Make data and code available when possible
Critical Evaluation Criteria
When Reviewing Scientific Work, Ask:
Validity Questions:
- Does the study measure what it claims to measure?
- Are the methods appropriate for the research question?
- Were controls adequate?
- Could confounding variables explain the results?
Reliability Questions:
- Are measurements consistent?
- Would the study produce similar results if repeated?
- Are inter-rater reliability and measurement precision reported?
Generalizability Questions:
- Is the sample representative of the target population?
- Are the conditions realistic or artificial?
- Do the results apply beyond the specific context?
Statistical Questions:
- Is the sample size adequate for the analysis?
- Are the statistical tests appropriate?
- Are effect sizes reported alongside p-values?
- Were multiple comparisons corrected?
Logical Questions:
- Do the conclusions follow from the data?
- Are alternative explanations considered?
- Are causal claims supported by the study design?
- Are limitations acknowledged?
Red Flags in Scientific Claims
- Cherry-picking data - Highlighting only supporting evidence
- Moving goalposts - Changing predictions after seeing results
- Ad hoc hypotheses - Adding explanations to rescue a failed prediction
- Appeal to authority - "Expert X says" without evidence
- Anecdotal evidence - Relying on personal stories over systematic data
- Correlation implies causation - Confusing association with causality
- Post hoc rationalization - Explaining results after the fact without prediction
- Ignoring base rates - Not considering prior probability
- Confirmation bias - Seeking only evidence that supports beliefs
- Publication bias - Only positive results get published
Standards for Causal Inference
Bradford Hill Criteria (adapted)
- Strength - Strong associations are more likely causal
- Consistency - Repeated observations by different researchers
- Specificity - Specific outcomes from specific causes
- Temporality - Cause precedes effect (essential)
- Biological gradient - Dose-response relationship
- Plausibility - Coherent with existing knowledge
- Coherence - Consistent with other evidence
- Experiment - Experimental evidence supports causation
- Analogy - Similar cause-effect relationships exist
Establishing Causation Requires:
- Temporal precedence (cause before effect)
- Covariation (cause and effect correlate)
- Elimination of alternative explanations
- Ideally: experimental manipulation showing cause produces effect
Peer Review and Scientific Consensus
Understanding Peer Review
- Filters obvious errors but isn't perfect
- Reviewers can miss problems or have biases
- Published ≠ proven; it means "passed initial scrutiny"
- Retraction mechanisms exist for flawed papers
Scientific Consensus
- Emerges from convergence of multiple independent lines of evidence
- Consensus can change with new evidence
- Individual studies rarely overturn consensus
- Consider the weight of evidence, not individual papers
Open Science Principles
Transparency Practices
- Preregistration of hypotheses and methods
- Open data sharing
- Open-source code
- Preprints for rapid dissemination
- Registered reports (peer review before data collection)
Why Transparency Matters
- Reduces publication bias
- Enables verification
- Prevents p-hacking and HARKing (Hypothesizing After Results are Known)
- Accelerates scientific progress