Scientific Critical Thinking
Overview
Critical thinking is a systematic process for evaluating scientific rigor. Assess methodology, experimental design, statistical validity, biases, confounding, and evidence quality using GRADE and Cochrane ROB frameworks. Apply this skill for critical analysis of scientific claims.
When to Use This Skill
This skill should be used when:
- Evaluating research methodology and experimental design
- Assessing statistical validity and evidence quality
- Identifying biases and confounding in studies
- Reviewing scientific claims and conclusions
- Conducting systematic reviews or meta-analyses
- Applying GRADE or Cochrane risk of bias assessments
- Providing critical analysis of research papers
Core Capabilities
1. Methodology Critique
Evaluate research methodology for rigor, validity, and potential flaws.
Apply when:
- Reviewing research papers
- Assessing experimental designs
- Evaluating study protocols
- Planning new research
Evaluation framework:
Study Design Assessment
- Is the design appropriate for the research question?
- Can the design support causal claims being made?
- Are comparison groups appropriate and adequate?
- Consider whether experimental, quasi-experimental, or observational design is justified
Validity Analysis
- Internal validity: Can we trust the causal inference?
- Check randomization quality
- Evaluate confounding control
- Assess selection bias
- Review attrition/dropout patterns
- External validity: Do results generalize?
- Evaluate sample representativeness
- Consider ecological validity of setting
- Assess whether conditions match target application
- Construct validity: Do measures capture intended constructs?
- Review measurement validation
- Check operational definitions
- Assess whether measures are direct or proxy
- Statistical conclusion validity: Are statistical inferences sound?
- Verify adequate power/sample size
- Check assumption compliance
- Evaluate test appropriateness
Control and Blinding
- Was randomization properly implemented (sequence generation, allocation concealment)?
- Was blinding feasible and implemented (participants, providers, assessors)?
- Are control conditions appropriate (placebo, active control, no treatment)?
- Could performance or detection bias affect results?
Measurement Quality
- Are instruments validated and reliable?
- Are measures objective when possible, or subjective with acknowledged limitations?
- Is outcome assessment standardized?
- Are multiple measures used to triangulate findings?
Reference: See references/scientific_method.md for detailed principles and references/experimental_design.md for comprehensive design checklist.
2. Bias Detection
Identify and evaluate potential sources of bias that could distort findings.
Apply when:
- Reviewing published research
- Designing new studies
- Interpreting conflicting evidence
- Assessing research quality
Systematic bias review:
Cognitive Biases (Researcher)
- Confirmation bias: Are only supporting findings highlighted?
- HARKing: Were hypotheses stated a priori or formed after seeing results?
- Publication bias: Are negative results missing from literature?
- Cherry-picking: Is evidence selectively reported?
- Check for preregistration and analysis plan transparency
Selection Biases
- Sampling bias: Is sample representative of target population?
- Volunteer bias: Do participants self-select in systematic ways?
- Attrition bias: Is dropout differential between groups?
- Survivorship bias: Are only "survivors" visible in sample?
- Examine participant flow diagrams and compare baseline characteristics
Measurement Biases
- Observer bias: Could expectations influence observations?
- Recall bias: Are retrospective reports systematically inaccurate?
- Social desirability: Are responses biased toward acceptability?
- Instrument bias: Do measurement tools systematically err?
- Evaluate blinding, validation, and measurement objectivity
Analysis Biases
- P-hacking: Were multiple analyses conducted until significance emerged?
- Outcome switching: Were non-significant outcomes replaced with significant ones?
- Selective reporting: Are all planned analyses reported?
- Subgroup fishing: Were subgroup analyses conducted without correction?
- Check for study registration and compare to published outcomes
Confounding
- What variables could affect both exposure and outcome?
- Were confounders measured and controlled (statistically or by design)?
- Could unmeasured confounding explain findings?
- Are there plausible alternative explanations?
Reference: See references/common_biases.md for comprehensive bias taxonomy with detection and mitigation strategies.
3. Statistical Analysis Evaluation
Critically assess statistical methods, interpretation, and reporting.
Apply when:
- Reviewing quantitative research
- Evaluating data-driven claims
- Assessing clinical trial results
- Reviewing meta-analyses
Statistical review checklist:
Sample Size and Power
- Was a priori power analysis conducted?
- Is sample adequate for detecting meaningful effects?
- Is the study underpowered (common problem)?
- Do significant results from small samples raise flags for inflated effect sizes?
Statistical Tests
- Are tests appropriate for data type and distribution?
- Were test assumptions checked and met?
- Are parametric tests justified, or should non-parametric alternatives be used?
- Is the analysis matched to study design (e.g., paired vs. independent)?
Multiple Comparisons
- Were multiple hypotheses tested?
- Was correction applied (Bonferroni, FDR, other)?
- Are primary outcomes distinguished from secondary/exploratory?
- Could findings be false positives from multiple testing?
P-Value Interpretation
- Are p-values interpreted correctly (probability of data if null is true)?
- Is non-significance incorrectly interpreted as "no effect"?
- Is statistical significance conflated with practical importance?
- Are exact p-values reported, or only "p < .05"?
- Is there suspicious clustering just below .05?
Effect Sizes and Confidence Intervals
- Are effect sizes reported alongside significance?
- Are confidence intervals provided to show precision?
- Is the effect size meaningful in practical terms?
- Are standardized effect sizes interpreted with field-specific context?
Missing Data
- How much data is missing?
- Is missing data mechanism considered (MCAR, MAR, MNAR)?
- How is missing data handled (deletion, imputation, maximum likelihood)?
- Could missing data bias results?
Regression and Modeling
- Is the model overfitted (too many predictors, no cross-validation)?
- Are predictions made outside the data range (extrapolation)?
- Are multicollinearity issues addressed?
- Are model assumptions checked?
Common Pitfalls
- Correlation treated as causation
- Ignoring regression to the mean
- Base rate neglect
- Texas sharpshooter fallacy (pattern finding in noise)
- Simpson's paradox (confounding by subgroups)
Reference: See references/statistical_pitfalls.md for detailed pitfalls and correct practices.
4. Evidence Quality Assessment
Evaluate the strength and quality of evidence systematically.
Apply when:
- Weighing evidence for decisions
- Conducting literature reviews
- Comparing conflicting findings
- Determining confidence in conclusions
Evidence evaluation framework:
Study Design Hierarchy
- Systematic reviews/meta-analyses (highest for intervention effects)
- Randomized controlled trials
- Cohort studies
- Case-control studies
- Cross-sectional studies
- Case series/reports
- Expert opinion (lowest)
Important: Higher-level designs aren't always better quality. A well-designed observational study can be stronger than a poorly-conducted RCT.
Quality Within Design Type
- Risk of bias assessment (use appropriate tool: Cochrane ROB, Newcastle-Ottawa, etc.)
- Methodological rigor
- Transparency and reporting completeness
- Conflicts of interest
GRADE Considerations (if applicable)
- Start with design type (RCT = high, observational = low)
- Downgrade for:
- Risk of bias
- Inconsistency across studies
- Indirectness (wrong population/intervention/outcome)
- Imprecision (wide confidence intervals, small samples)
- Publication bias
- Upgrade for:
- Large effect sizes
- Dose-response relationships
- Confounders would reduce (not increase) effect
Convergence of Evidence
- Stronger when:
- Multiple independent replications
- Different research groups and settings
- Different methodologies converge on same conclusion
- Mechanistic and empirical evidence align
- Weaker when:
- Single study or research group
- Contradictory findings in literature
- Publication bias evident
- No replication attempts
Contextual Factors
- Biological/theoretical plausibility
- Consistency with established knowledge
- Temporality (cause precedes effect)
- Specificity of relationship
- Strength of association
Reference: See references/evidence_hierarchy.md for detailed hierarchy, GRADE system, and quality assessment tools.
5. Logical Fallacy Identification
Detect and name logical errors in scientific arguments and claims.
Apply when:
- Evaluating scientific claims
- Reviewing discussion/conclusion sections
- Assessing popular science communication
- Identifying flawed reasoning
Common fallacies in science:
Causation Fallacies
- Post hoc ergo propter hoc: "B followed A, so A caused B"
- Correlation = causation: Confusing association with causality
- Reverse causation: Mistaking cause for effect
- Single cause fallacy: Attributing complex outcomes to one factor
Generalization Fallacies
- Hasty generalization: Broad conclusions from small samples
- Anecdotal fallacy: Personal stories as proof
- Cherry-picking: Selecting only supporting evidence
- Ecological fallacy: Group patterns applied to individuals
Authority and Source Fallacies
- Appeal to authority: "Expert said it, so it's true" (without evidence)
- Ad hominem: Attacking person, not argument
- Genetic fallacy: Judging by origin, not merits
- Appeal to nature: "Natural = good/safe"
Statistical Fallacies
- Base rate neglect: Ignoring prior probability
- Texas sharpshooter: Finding patterns in random data
- Multiple comparisons: Not correcting for multiple tests
- Prosecutor's fallacy: Confusing P(E|H) with P(H|E)
Structural Fallacies
- False dichotomy: "Either A or B" when more options exist
- Moving goalposts: Changing evidence standards after they're met
- Begging the question: Circular reasoning
- Straw man: Misrepresenting arguments to attack them
Science-Specific Fallacies
- Galileo gambit: "They laughed at Galileo, so my fringe idea is correct"
- Argument from ignorance: "Not proven false, so true"
- Nirvana fallacy: Rejecting imperfect solutions
- Unfalsifiability: Making untestable claims
When identifying fallacies:
- Name the specific fallacy
- Explain why the reasoning is flawed
- Identify what evidence would be needed for valid inference
- Note that fallacious reasoning doesn't prove the conclusion false—just that this argument doesn't support it
Reference: See references/logical_fallacies.md for comprehensive fallacy catalog with examples and detection strategies.
6. Research Design Guidance
Provide constructive guidance for planning rigorous studies.
Apply when:
- Helping design new experiments
- Planning research projects
- Reviewing research proposals
- Improving study protocols
Design process:
Research Question Refinement
- Ensure question is specific, answerable, and falsifiable
- Verify it addresses a gap or contradiction in literature
- Confirm feasibility (resources, ethics, time)
- Define variables operationally
Design Selection
- Match design to question (causal → experimental; associational → observational)
- Consider feasibility and ethical constraints
- Choose between-subjects, within-subjects, or mixed designs
- Plan factorial designs if testing multiple factors
Bias Minimization Strategy
- Implement randomization when possible
- Plan blinding at all feasible levels (participants, providers, assessors)
- Identify and plan to control confounds (randomization, matching, stratification, statistical adjustment)
- Standardize all procedures
- Plan to minimize attrition
Sample Planning
- Conduct a priori power analysis (specify expected effect, desired power, alpha)
- Account for attrition in sample size
- Define clear inclusion/exclusion criteria
- Consider recruitment strategy and feasibility
- Plan for sample representativeness
Measurement Strategy
- Select validated, reliable instruments
- Use objective measures when possible
- Plan multiple measures of key constructs (triangulation)
- Ensure measures are sensitive to expected changes
- Establish inter-rater reliability procedures
Analysis Planning
- Prespecify all hypotheses and analyses
- Designate primary outcome clearly
- Plan statistical tests with assumption checks
- Specify how missing data will be handled
- Plan to report effect sizes and confidence intervals
- Consider multiple comparison corrections
Transparency and Rigor
- Preregister study and analysis plan
- Use reporting guidelines (CONSORT, STROBE, PRISMA)
- Plan to report all outcomes, not just significant ones
- Distinguish confirmatory from exploratory analyses
- Commit to data/code sharing
Reference: See references/experimental_design.md for comprehensive design checklist covering all stages from question to dissemination.
7. Claim Evaluation
Systematically evaluate scientific claims for validity and support.
Apply when:
- Assessing conclusions in papers
- Evaluating media reports of research
- Reviewing abstract or introduction claims
- Checking if data support conclusions
Claim evaluation process:
Identify the Claim
- What exactly is being claimed?
- Is it a causal claim, associational claim, or descriptive claim?
- How strong is the claim (proven, likely, suggested, possible)?
Assess the Evidence
- What evidence is provided?
- Is evidence direct or indirect?
- Is evidence sufficient for the strength of claim?
- Are alternative explanations ruled out?
Check Logical Connection
- Do conclusions follow from the data?
- Are there logical leaps?
- Is correlational data used to support causal claims?
- Are limitations acknowledged?
Evaluate Proportionality
- Is confidence proportional to evidence strength?
- Are hedging words used appropriately?
- Are limitations downplayed?
- Is speculation clearly labeled?
Check for Overgeneralization
- Do claims extend beyond the sample studied?
- Are population restrictions acknowledged?
- Is context-dependence recognized?
- Are caveats about generalization included?
Red Flags
- Causal language from correlational studies
- "Proves" or absolute certainty
- Cherry-picked citations
- Ignoring contradictory evidence
- Dismissing limitations
- Extrapolation beyond data
Provide specific feedback:
- Quote the problematic claim
- Explain what evidence would be needed to support it
- Suggest appropriate hedging language if warranted
- Distinguish between data (what was found) and interpretation (what it means)
Application Guidelines
General Approach
Be Constructive
- Identify strengths as well as weaknesses
- Suggest improvements rather than just criticizing
- Distinguish between fatal flaws and minor limitations
- Recognize that all research has limitations
Be Specific
- Point to specific instances (e.g., "Table 2 shows..." or "In the Methods section...")
- Quote problematic statements
- Provide concrete examples of issues
- Reference specific principles or standards violated
Be Proportionate
- Match criticism severity to issue importance
- Distinguish between major threats to validity and minor concerns
- Consider whether issues affect primary conclusions
- Acknowledge uncertainty in your own assessments
Apply Consistent Standards
- Use same criteria across all studies
- Don't apply stricter standards to findings you dislike
- Acknowledge your own potential biases
- Base judgments on methodology, not results
Consider Context
- Acknowledge practical and ethical constraints
- Consider field-specific norms for effect sizes and methods
- Recognize exploratory vs. confirmatory contexts
- Account for resource limitations in evaluating studies
When Providing Critique
Structure feedback as:
- Summary: Brief overview of what was evaluated
- Strengths: What was done well (important for credibility and learning)
- Concerns: Issues organized by severity
- Critical issues (threaten validity of main conclusions)
- Important issues (affect interpretation but not fatally)
- Minor issues (worth noting but don't change conclusions)
- Specific Recommendations: Actionable suggestions for improvement
- Overall Assessment: Balanced conclusion about evidence quality and what can be concluded
Use precise terminology:
- Name specific biases, fallacies, and methodological issues
- Reference established standards and guidelines
- Cite principles from scientific methodology
- Use technical terms accurately
When Uncertain
- Acknowledge uncertainty: "This could be X or Y; additional information needed is Z"
- Ask clarifying questions: "Was [methodological detail] done? This affects interpretation."
- Provide conditional assessments: "If X was done, then Y follows; if not, then Z is concern"
- Note what additional information would resolve uncertainty
Reference Materials
This skill includes comprehensive reference materials that provide detailed frameworks for critical evaluation:
references/scientific_method.md - Core principles of scientific methodology, the scientific process, critical evaluation criteria, red flags in scientific claims, causal inference standards, peer review, and open science principles
references/common_biases.md - Comprehensive taxonomy of cognitive, experimental, methodological, statistical, and analysis biases with detection and mitigation strategies
references/statistical_pitfalls.md - Common statistical errors and misinterpretations including p-value misunderstandings, multiple comparisons problems, sample size issues, effect size mistakes, correlation/causation confusion, regression pitfalls, and meta-analysis issues
references/evidence_hierarchy.md - Traditional evidence hierarchy, GRADE system, study quality assessment criteria, domain-specific considerations, evidence synthesis principles, and practical decision frameworks
references/logical_fallacies.md - Logical fallacies common in scientific discourse organized by type (causation, generalization, authority, relevance, structure, statistical) with examples and detection strategies
references/experimental_design.md - Comprehensive experimental design checklist covering research questions, hypotheses, study design selection, variables, sampling, blinding, randomization, control groups, procedures, measurement, bias minimization, data management, statistical planning, ethical considerations, validity threats, and reporting standards
When to consult references:
- Load references into context when detailed frameworks are needed
- Use grep to search references for specific topics:
grep -r "pattern" references/
- References provide depth; SKILL.md provides procedural guidance
- Consult references for comprehensive lists, detailed criteria, and specific examples
Remember
Scientific critical thinking is about:
- Systematic evaluation using established principles
- Constructive critique that improves science
- Proportional confidence to evidence strength
- Transparency about uncertainty and limitations
- Consistent application of standards
- Recognition that all research has limitations
- Balance between skepticism and openness to evidence
Always distinguish between:
- Data (what was observed) and interpretation (what it means)
- Correlation and causation
- Statistical significance and practical importance
- Exploratory and confirmatory findings
- What is known and what is uncertain
- Evidence against a claim and evidence for the null
Goals of critical thinking:
- Identify strengths and weaknesses accurately
- Determine what conclusions are supported
- Recognize limitations and uncertainties
- Suggest improvements for future work
- Advance scientific understanding
1---2name: "scientific-critical-thinking"3description: "Evaluate research rigor. Assess methodology, experimental design, statistical validity, biases, confounding, evidence quality (GRADE, Cochrane ROB), for critical analysis of scientific claims."4metadata:5 source: https://skillmd.com/skills/viggyv/scientific-critical-thinking6---78# Scientific Critical Thinking910## Overview1112Critical thinking is a systematic process for evaluating scientific rigor. Assess methodology, experimental design, statistical validity, biases, confounding, and evidence quality using GRADE and Cochrane ROB frameworks. Apply this skill for critical analysis of scientific claims.1314## When to Use This Skill1516This skill should be used when:17- Evaluating research methodology and experimental design18- Assessing statistical validity and evidence quality19- Identifying biases and confounding in studies20- Reviewing scientific claims and conclusions21- Conducting systematic reviews or meta-analyses22- Applying GRADE or Cochrane risk of bias assessments23- Providing critical analysis of research papers2425## Core Capabilities2627### 1. Methodology Critique2829Evaluate research methodology for rigor, validity, and potential flaws.3031**Apply when:**32- Reviewing research papers33- Assessing experimental designs34- Evaluating study protocols35- Planning new research3637**Evaluation framework:**38391. **Study Design Assessment**40 - Is the design appropriate for the research question?41 - Can the design support causal claims being made?42 - Are comparison groups appropriate and adequate?43 - Consider whether experimental, quasi-experimental, or observational design is justified44452. **Validity Analysis**46 - **Internal validity:** Can we trust the causal inference?47 - Check randomization quality48 - Evaluate confounding control49 - Assess selection bias50 - Review attrition/dropout patterns51 - **External validity:** Do results generalize?52 - Evaluate sample representativeness53 - Consider ecological validity of setting54 - Assess whether conditions match target application55 - **Construct validity:** Do measures capture intended constructs?56 - Review measurement validation57 - Check operational definitions58 - Assess whether measures are direct or proxy59 - **Statistical conclusion validity:** Are statistical inferences sound?60 - Verify adequate power/sample size61 - Check assumption compliance62 - Evaluate test appropriateness63643. **Control and Blinding**65 - Was randomization properly implemented (sequence generation, allocation concealment)?66 - Was blinding feasible and implemented (participants, providers, assessors)?67 - Are control conditions appropriate (placebo, active control, no treatment)?68 - Could performance or detection bias affect results?69704. **Measurement Quality**71 - Are instruments validated and reliable?72 - Are measures objective when possible, or subjective with acknowledged limitations?73 - Is outcome assessment standardized?74 - Are multiple measures used to triangulate findings?7576**Reference:** See `references/scientific_method.md` for detailed principles and `references/experimental_design.md` for comprehensive design checklist.7778### 2. Bias Detection7980Identify and evaluate potential sources of bias that could distort findings.8182**Apply when:**83- Reviewing published research84- Designing new studies85- Interpreting conflicting evidence86- Assessing research quality8788**Systematic bias review:**89901. **Cognitive Biases (Researcher)**91 - **Confirmation bias:** Are only supporting findings highlighted?92 - **HARKing:** Were hypotheses stated a priori or formed after seeing results?93 - **Publication bias:** Are negative results missing from literature?94 - **Cherry-picking:** Is evidence selectively reported?95 - Check for preregistration and analysis plan transparency96972. **Selection Biases**98 - **Sampling bias:** Is sample representative of target population?99 - **Volunteer bias:** Do participants self-select in systematic ways?100 - **Attrition bias:** Is dropout differential between groups?101 - **Survivorship bias:** Are only "survivors" visible in sample?102 - Examine participant flow diagrams and compare baseline characteristics1031043. **Measurement Biases**105 - **Observer bias:** Could expectations influence observations?106 - **Recall bias:** Are retrospective reports systematically inaccurate?107 - **Social desirability:** Are responses biased toward acceptability?108 - **Instrument bias:** Do measurement tools systematically err?109 - Evaluate blinding, validation, and measurement objectivity1101114. **Analysis Biases**112 - **P-hacking:** Were multiple analyses conducted until significance emerged?113 - **Outcome switching:** Were non-significant outcomes replaced with significant ones?114 - **Selective reporting:** Are all planned analyses reported?115 - **Subgroup fishing:** Were subgroup analyses conducted without correction?116 - Check for study registration and compare to published outcomes1171185. **Confounding**119 - What variables could affect both exposure and outcome?120 - Were confounders measured and controlled (statistically or by design)?121 - Could unmeasured confounding explain findings?122 - Are there plausible alternative explanations?123124**Reference:** See `references/common_biases.md` for comprehensive bias taxonomy with detection and mitigation strategies.125126### 3. Statistical Analysis Evaluation127128Critically assess statistical methods, interpretation, and reporting.129130**Apply when:**131- Reviewing quantitative research132- Evaluating data-driven claims133- Assessing clinical trial results134- Reviewing meta-analyses135136**Statistical review checklist:**1371381. **Sample Size and Power**139 - Was a priori power analysis conducted?140 - Is sample adequate for detecting meaningful effects?141 - Is the study underpowered (common problem)?142 - Do significant results from small samples raise flags for inflated effect sizes?1431442. **Statistical Tests**145 - Are tests appropriate for data type and distribution?146 - Were test assumptions checked and met?147 - Are parametric tests justified, or should non-parametric alternatives be used?148 - Is the analysis matched to study design (e.g., paired vs. independent)?1491503. **Multiple Comparisons**151 - Were multiple hypotheses tested?152 - Was correction applied (Bonferroni, FDR, other)?153 - Are primary outcomes distinguished from secondary/exploratory?154 - Could findings be false positives from multiple testing?1551564. **P-Value Interpretation**157 - Are p-values interpreted correctly (probability of data if null is true)?158 - Is non-significance incorrectly interpreted as "no effect"?159 - Is statistical significance conflated with practical importance?160 - Are exact p-values reported, or only "p < .05"?161 - Is there suspicious clustering just below .05?1621635. **Effect Sizes and Confidence Intervals**164 - Are effect sizes reported alongside significance?165 - Are confidence intervals provided to show precision?166 - Is the effect size meaningful in practical terms?167 - Are standardized effect sizes interpreted with field-specific context?1681696. **Missing Data**170 - How much data is missing?171 - Is missing data mechanism considered (MCAR, MAR, MNAR)?172 - How is missing data handled (deletion, imputation, maximum likelihood)?173 - Could missing data bias results?1741757. **Regression and Modeling**176 - Is the model overfitted (too many predictors, no cross-validation)?177 - Are predictions made outside the data range (extrapolation)?178 - Are multicollinearity issues addressed?179 - Are model assumptions checked?1801818. **Common Pitfalls**182 - Correlation treated as causation183 - Ignoring regression to the mean184 - Base rate neglect185 - Texas sharpshooter fallacy (pattern finding in noise)186 - Simpson's paradox (confounding by subgroups)187188**Reference:** See `references/statistical_pitfalls.md` for detailed pitfalls and correct practices.189190### 4. Evidence Quality Assessment191192Evaluate the strength and quality of evidence systematically.193194**Apply when:**195- Weighing evidence for decisions196- Conducting literature reviews197- Comparing conflicting findings198- Determining confidence in conclusions199200**Evidence evaluation framework:**2012021. **Study Design Hierarchy**203 - Systematic reviews/meta-analyses (highest for intervention effects)204 - Randomized controlled trials205 - Cohort studies206 - Case-control studies207 - Cross-sectional studies208 - Case series/reports209 - Expert opinion (lowest)210211 **Important:** Higher-level designs aren't always better quality. A well-designed observational study can be stronger than a poorly-conducted RCT.2122132. **Quality Within Design Type**214 - Risk of bias assessment (use appropriate tool: Cochrane ROB, Newcastle-Ottawa, etc.)215 - Methodological rigor216 - Transparency and reporting completeness217 - Conflicts of interest2182193. **GRADE Considerations (if applicable)**220 - Start with design type (RCT = high, observational = low)221 - **Downgrade for:**222 - Risk of bias223 - Inconsistency across studies224 - Indirectness (wrong population/intervention/outcome)225 - Imprecision (wide confidence intervals, small samples)226 - Publication bias227 - **Upgrade for:**228 - Large effect sizes229 - Dose-response relationships230 - Confounders would reduce (not increase) effect2312324. **Convergence of Evidence**233 - **Stronger when:**234 - Multiple independent replications235 - Different research groups and settings236 - Different methodologies converge on same conclusion237 - Mechanistic and empirical evidence align238 - **Weaker when:**239 - Single study or research group240 - Contradictory findings in literature241 - Publication bias evident242 - No replication attempts2432445. **Contextual Factors**245 - Biological/theoretical plausibility246 - Consistency with established knowledge247 - Temporality (cause precedes effect)248 - Specificity of relationship249 - Strength of association250251**Reference:** See `references/evidence_hierarchy.md` for detailed hierarchy, GRADE system, and quality assessment tools.252253### 5. Logical Fallacy Identification254255Detect and name logical errors in scientific arguments and claims.256257**Apply when:**258- Evaluating scientific claims259- Reviewing discussion/conclusion sections260- Assessing popular science communication261- Identifying flawed reasoning262263**Common fallacies in science:**2642651. **Causation Fallacies**266 - **Post hoc ergo propter hoc:** "B followed A, so A caused B"267 - **Correlation = causation:** Confusing association with causality268 - **Reverse causation:** Mistaking cause for effect269 - **Single cause fallacy:** Attributing complex outcomes to one factor2702712. **Generalization Fallacies**272 - **Hasty generalization:** Broad conclusions from small samples273 - **Anecdotal fallacy:** Personal stories as proof274 - **Cherry-picking:** Selecting only supporting evidence275 - **Ecological fallacy:** Group patterns applied to individuals2762773. **Authority and Source Fallacies**278 - **Appeal to authority:** "Expert said it, so it's true" (without evidence)279 - **Ad hominem:** Attacking person, not argument280 - **Genetic fallacy:** Judging by origin, not merits281 - **Appeal to nature:** "Natural = good/safe"2822834. **Statistical Fallacies**284 - **Base rate neglect:** Ignoring prior probability285 - **Texas sharpshooter:** Finding patterns in random data286 - **Multiple comparisons:** Not correcting for multiple tests287 - **Prosecutor's fallacy:** Confusing P(E|H) with P(H|E)2882895. **Structural Fallacies**290 - **False dichotomy:** "Either A or B" when more options exist291 - **Moving goalposts:** Changing evidence standards after they're met292 - **Begging the question:** Circular reasoning293 - **Straw man:** Misrepresenting arguments to attack them2942956. **Science-Specific Fallacies**296 - **Galileo gambit:** "They laughed at Galileo, so my fringe idea is correct"297 - **Argument from ignorance:** "Not proven false, so true"298 - **Nirvana fallacy:** Rejecting imperfect solutions299 - **Unfalsifiability:** Making untestable claims300301**When identifying fallacies:**302- Name the specific fallacy303- Explain why the reasoning is flawed304- Identify what evidence would be needed for valid inference305- Note that fallacious reasoning doesn't prove the conclusion false—just that this argument doesn't support it306307**Reference:** See `references/logical_fallacies.md` for comprehensive fallacy catalog with examples and detection strategies.308309### 6. Research Design Guidance310311Provide constructive guidance for planning rigorous studies.312313**Apply when:**314- Helping design new experiments315- Planning research projects316- Reviewing research proposals317- Improving study protocols318319**Design process:**3203211. **Research Question Refinement**322 - Ensure question is specific, answerable, and falsifiable323 - Verify it addresses a gap or contradiction in literature324 - Confirm feasibility (resources, ethics, time)325 - Define variables operationally3263272. **Design Selection**328 - Match design to question (causal → experimental; associational → observational)329 - Consider feasibility and ethical constraints330 - Choose between-subjects, within-subjects, or mixed designs331 - Plan factorial designs if testing multiple factors3323333. **Bias Minimization Strategy**334 - Implement randomization when possible335 - Plan blinding at all feasible levels (participants, providers, assessors)336 - Identify and plan to control confounds (randomization, matching, stratification, statistical adjustment)337 - Standardize all procedures338 - Plan to minimize attrition3393404. **Sample Planning**341 - Conduct a priori power analysis (specify expected effect, desired power, alpha)342 - Account for attrition in sample size343 - Define clear inclusion/exclusion criteria344 - Consider recruitment strategy and feasibility345 - Plan for sample representativeness3463475. **Measurement Strategy**348 - Select validated, reliable instruments349 - Use objective measures when possible350 - Plan multiple measures of key constructs (triangulation)351 - Ensure measures are sensitive to expected changes352 - Establish inter-rater reliability procedures3533546. **Analysis Planning**355 - Prespecify all hypotheses and analyses356 - Designate primary outcome clearly357 - Plan statistical tests with assumption checks358 - Specify how missing data will be handled359 - Plan to report effect sizes and confidence intervals360 - Consider multiple comparison corrections3613627. **Transparency and Rigor**363 - Preregister study and analysis plan364 - Use reporting guidelines (CONSORT, STROBE, PRISMA)365 - Plan to report all outcomes, not just significant ones366 - Distinguish confirmatory from exploratory analyses367 - Commit to data/code sharing368369**Reference:** See `references/experimental_design.md` for comprehensive design checklist covering all stages from question to dissemination.370371### 7. Claim Evaluation372373Systematically evaluate scientific claims for validity and support.374375**Apply when:**376- Assessing conclusions in papers377- Evaluating media reports of research378- Reviewing abstract or introduction claims379- Checking if data support conclusions380381**Claim evaluation process:**3823831. **Identify the Claim**384 - What exactly is being claimed?385 - Is it a causal claim, associational claim, or descriptive claim?386 - How strong is the claim (proven, likely, suggested, possible)?3873882. **Assess the Evidence**389 - What evidence is provided?390 - Is evidence direct or indirect?391 - Is evidence sufficient for the strength of claim?392 - Are alternative explanations ruled out?3933943. **Check Logical Connection**395 - Do conclusions follow from the data?396 - Are there logical leaps?397 - Is correlational data used to support causal claims?398 - Are limitations acknowledged?3994004. **Evaluate Proportionality**401 - Is confidence proportional to evidence strength?402 - Are hedging words used appropriately?403 - Are limitations downplayed?404 - Is speculation clearly labeled?4054065. **Check for Overgeneralization**407 - Do claims extend beyond the sample studied?408 - Are population restrictions acknowledged?409 - Is context-dependence recognized?410 - Are caveats about generalization included?4114126. **Red Flags**413 - Causal language from correlational studies414 - "Proves" or absolute certainty415 - Cherry-picked citations416 - Ignoring contradictory evidence417 - Dismissing limitations418 - Extrapolation beyond data419420**Provide specific feedback:**421- Quote the problematic claim422- Explain what evidence would be needed to support it423- Suggest appropriate hedging language if warranted424- Distinguish between data (what was found) and interpretation (what it means)425426## Application Guidelines427428### General Approach4294301. **Be Constructive**431 - Identify strengths as well as weaknesses432 - Suggest improvements rather than just criticizing433 - Distinguish between fatal flaws and minor limitations434 - Recognize that all research has limitations4354362. **Be Specific**437 - Point to specific instances (e.g., "Table 2 shows..." or "In the Methods section...")438 - Quote problematic statements439 - Provide concrete examples of issues440 - Reference specific principles or standards violated4414423. **Be Proportionate**443 - Match criticism severity to issue importance444 - Distinguish between major threats to validity and minor concerns445 - Consider whether issues affect primary conclusions446 - Acknowledge uncertainty in your own assessments4474484. **Apply Consistent Standards**449 - Use same criteria across all studies450 - Don't apply stricter standards to findings you dislike451 - Acknowledge your own potential biases452 - Base judgments on methodology, not results4534545. **Consider Context**455 - Acknowledge practical and ethical constraints456 - Consider field-specific norms for effect sizes and methods457 - Recognize exploratory vs. confirmatory contexts458 - Account for resource limitations in evaluating studies459460### When Providing Critique461462**Structure feedback as:**4634641. **Summary:** Brief overview of what was evaluated4652. **Strengths:** What was done well (important for credibility and learning)4663. **Concerns:** Issues organized by severity467 - Critical issues (threaten validity of main conclusions)468 - Important issues (affect interpretation but not fatally)469 - Minor issues (worth noting but don't change conclusions)4704. **Specific Recommendations:** Actionable suggestions for improvement4715. **Overall Assessment:** Balanced conclusion about evidence quality and what can be concluded472473**Use precise terminology:**474- Name specific biases, fallacies, and methodological issues475- Reference established standards and guidelines476- Cite principles from scientific methodology477- Use technical terms accurately478479### When Uncertain480481- **Acknowledge uncertainty:** "This could be X or Y; additional information needed is Z"482- **Ask clarifying questions:** "Was [methodological detail] done? This affects interpretation."483- **Provide conditional assessments:** "If X was done, then Y follows; if not, then Z is concern"484- **Note what additional information would resolve uncertainty**485486## Reference Materials487488This skill includes comprehensive reference materials that provide detailed frameworks for critical evaluation:489490- **`references/scientific_method.md`** - Core principles of scientific methodology, the scientific process, critical evaluation criteria, red flags in scientific claims, causal inference standards, peer review, and open science principles491492- **`references/common_biases.md`** - Comprehensive taxonomy of cognitive, experimental, methodological, statistical, and analysis biases with detection and mitigation strategies493494- **`references/statistical_pitfalls.md`** - Common statistical errors and misinterpretations including p-value misunderstandings, multiple comparisons problems, sample size issues, effect size mistakes, correlation/causation confusion, regression pitfalls, and meta-analysis issues495496- **`references/evidence_hierarchy.md`** - Traditional evidence hierarchy, GRADE system, study quality assessment criteria, domain-specific considerations, evidence synthesis principles, and practical decision frameworks497498- **`references/logical_fallacies.md`** - Logical fallacies common in scientific discourse organized by type (causation, generalization, authority, relevance, structure, statistical) with examples and detection strategies499500- **`references/experimental_design.md`** - Comprehensive experimental design checklist covering research questions, hypotheses, study design selection, variables, sampling, blinding, randomization, control groups, procedures, measurement, bias minimization, data management, statistical planning, ethical considerations, validity threats, and reporting standards501502**When to consult references:**503- Load references into context when detailed frameworks are needed504- Use grep to search references for specific topics: `grep -r "pattern" references/`505- References provide depth; SKILL.md provides procedural guidance506- Consult references for comprehensive lists, detailed criteria, and specific examples507508## Remember509510**Scientific critical thinking is about:**511- Systematic evaluation using established principles512- Constructive critique that improves science513- Proportional confidence to evidence strength514- Transparency about uncertainty and limitations515- Consistent application of standards516- Recognition that all research has limitations517- Balance between skepticism and openness to evidence518519**Always distinguish between:**520- Data (what was observed) and interpretation (what it means)521- Correlation and causation522- Statistical significance and practical importance523- Exploratory and confirmatory findings524- What is known and what is uncertain525- Evidence against a claim and evidence for the null526527**Goals of critical thinking:**5281. Identify strengths and weaknesses accurately5292. Determine what conclusions are supported5303. Recognize limitations and uncertainties5314. Suggest improvements for future work5325. Advance scientific understanding