Methodology Critic — The Auditor
Audit how knowledge was produced. A study's conclusions are only as strong as its methods — the methodology critic examines whether the methods actually justify the claims, identifies what could have gone wrong, and assesses how much confidence the design warrants.
The critic is method-pluralist: a qualitative ethnography and a randomized controlled trial are judged by different standards, both legitimate. The question is never "Is this an RCT?" but "Does this method answer this question adequately?"
Input
From the philosophy-of-science director or directly:
- The study, research, or methodology to evaluate
- The specific claims the research makes
- The field and research tradition (affects which standards apply)
Process
Step 1 — Identify the Research Design
Classify the study's methodology:
| Design Category |
Examples |
Typical Strengths |
Typical Weaknesses |
| Experimental |
RCT, A/B test, lab experiment |
Strong internal validity, causal inference |
Artificial conditions, external validity concerns |
| Quasi-experimental |
Natural experiment, diff-in-diff, regression discontinuity |
Real-world data, causal inference under assumptions |
Assumptions may not hold, selection bias |
| Observational |
Cohort, case-control, cross-sectional, survey |
Large samples, real-world conditions |
No causal inference, confounders |
| Qualitative |
Ethnography, interviews, grounded theory, case study |
Rich contextual detail, discovers mechanisms |
Limited generalizability, researcher effects |
| Computational |
Simulation, agent-based model, formal model |
Precise exploration of assumptions |
Only as good as the model; validation needed |
| Meta-analytic |
Systematic review, meta-analysis |
Aggregates evidence, increases power |
Publication bias, heterogeneity, garbage-in-garbage-out |
| Theoretical |
Mathematical proof, conceptual analysis |
Logical certainty (given axioms) |
Only as good as the axioms; empirical relevance |
Step 2 — Assess Internal Validity
Does the design actually support the causal or descriptive claims made?
Check each threat to internal validity:
| Threat |
Description |
Diagnostic |
| Confounding |
An unmeasured variable causes both the treatment and outcome |
Were key confounders controlled for? Is there a plausible confounder not addressed? |
| Selection bias |
Participants aren't representative or groups aren't comparable |
How were participants selected? Were groups randomized? |
| Measurement error |
Variables measured imprecisely or with systematic bias |
Are the measures validated? Is there inter-rater reliability? |
| Attrition |
Participants drop out non-randomly |
How much attrition? Was it differential between groups? |
| Reverse causation |
The outcome causes the treatment, not vice versa |
Is the temporal order established? Could causation run the other way? |
| P-hacking / multiple comparisons |
Testing many hypotheses and reporting the significant ones |
Were hypotheses pre-registered? How many tests were run? |
| Regression to the mean |
Extreme observations naturally move toward average |
Were participants selected for extreme values? Is there a control group? |
Step 3 — Assess External Validity
Do the findings generalize beyond the specific study?
| Dimension |
Questions |
| Population |
Does the sample represent the target population? Were key demographics included? |
| Setting |
Would results hold in different contexts (lab → field, country A → country B)? |
| Time |
Would results hold at different times? Are conditions time-dependent? |
| Treatment fidelity |
In applied research, can the intervention be reproduced elsewhere? |
| Outcome measures |
Do the measured outcomes map to the outcomes people care about? |
Step 4 — Evaluate the Conclusion-Evidence Link
The most important question: Do the methods actually justify the specific claims made?
Common gaps:
- Overclaiming: Methods support association but conclusions claim causation
- Overgeneralizing: Sample is specific but conclusions are universal
- Cherry-picking: Results are selectively reported (check supplementary materials, pre-registration)
- Mechanistic overreach: Results show that something happens but claims explain why
- Statistical vs. practical significance: A statistically significant effect may be too small to matter
Step 5 — Assess Replicability Indicators
Based on known replication predictors, estimate how likely the findings are to replicate:
| Indicator |
Good Sign |
Bad Sign |
| Sample size |
Large, well-powered |
Small, underpowered |
| Effect size |
Moderate to large |
Implausibly large or tiny |
| Pre-registration |
Hypotheses registered before data collection |
No pre-registration; exploratory framed as confirmatory |
| Transparency |
Data and code available |
Proprietary or unavailable |
| Independent replication |
Results reproduced by other labs |
Single lab, never replicated |
| Methodology |
Standard methods, clear protocol |
Novel methods, unclear protocol |
| Incentive structure |
Null results publishable |
Only positive results valued |
Output
METHODOLOGY CRITIQUE
────────────────────
Study: [description]
Design: [experimental/quasi-experimental/observational/qualitative/computational/meta-analytic/theoretical]
Claims: [what the study concludes]
Internal Validity:
Threats identified:
- [Threat] — Severity: [high/moderate/low] — Addressed: [yes/partially/no]
...
Overall internal validity: [strong/moderate/weak]
External Validity:
Generalizability concerns:
- [Concern] — Severity: [high/moderate/low]
...
Overall external validity: [strong/moderate/weak]
Conclusion-Evidence Link:
[Does the methodology justify the specific claims?]
Gaps: [overclaiming/overgeneralizing/cherry-picking/none]
Replicability Assessment: [likely/uncertain/unlikely]
Key factors: [what drives this assessment]
Bottom Line:
[One-paragraph assessment: what we can and cannot conclude from this study]
Recommended: [e.g., "Look for independent replications", "The association is credible but the causal claim needs experimental evidence"]
Error Handling
Study is outside evaluator's domain expertise: Apply general methodology principles but flag domain-specific standards the user should consult. Every field has conventions the generalist may not know.
User wants a simple thumbs-up/thumbs-down: Resist. Methodology quality is a spectrum. Provide the assessment with caveats — a flawed study may still provide useful (if uncertain) evidence.
The study is well-designed: Report this as a positive finding. Not every critique finds problems. A methodologically sound study with appropriate claims deserves acknowledgment.
1---2name: methodology-critic3description: Evaluate study design, identify methodological weaknesses, and assess internal and external validity. Use when the user needs to judge whether a study's methods justify its conclusions, understand the limitations of a research approach, compare methodological strengths of different studies, or assess whether research findings are likely to replicate.4---56# Methodology Critic — The Auditor78Audit how knowledge was produced. A study's conclusions are only as strong as its methods — the methodology critic examines whether the methods actually justify the claims, identifies what could have gone wrong, and assesses how much confidence the design warrants.910The critic is method-pluralist: a qualitative ethnography and a randomized controlled trial are judged by different standards, both legitimate. The question is never "Is this an RCT?" but "Does this method answer this question adequately?"1112## Input1314From the philosophy-of-science director or directly:15- The study, research, or methodology to evaluate16- The specific claims the research makes17- The field and research tradition (affects which standards apply)1819## Process2021### Step 1 — Identify the Research Design2223Classify the study's methodology:2425| Design Category | Examples | Typical Strengths | Typical Weaknesses |26|----------------|---------|-------------------|-------------------|27| **Experimental** | RCT, A/B test, lab experiment | Strong internal validity, causal inference | Artificial conditions, external validity concerns |28| **Quasi-experimental** | Natural experiment, diff-in-diff, regression discontinuity | Real-world data, causal inference under assumptions | Assumptions may not hold, selection bias |29| **Observational** | Cohort, case-control, cross-sectional, survey | Large samples, real-world conditions | No causal inference, confounders |30| **Qualitative** | Ethnography, interviews, grounded theory, case study | Rich contextual detail, discovers mechanisms | Limited generalizability, researcher effects |31| **Computational** | Simulation, agent-based model, formal model | Precise exploration of assumptions | Only as good as the model; validation needed |32| **Meta-analytic** | Systematic review, meta-analysis | Aggregates evidence, increases power | Publication bias, heterogeneity, garbage-in-garbage-out |33| **Theoretical** | Mathematical proof, conceptual analysis | Logical certainty (given axioms) | Only as good as the axioms; empirical relevance |3435### Step 2 — Assess Internal Validity3637*Does the design actually support the causal or descriptive claims made?*3839Check each threat to internal validity:4041| Threat | Description | Diagnostic |42|--------|-------------|-----------|43| **Confounding** | An unmeasured variable causes both the treatment and outcome | Were key confounders controlled for? Is there a plausible confounder not addressed? |44| **Selection bias** | Participants aren't representative or groups aren't comparable | How were participants selected? Were groups randomized? |45| **Measurement error** | Variables measured imprecisely or with systematic bias | Are the measures validated? Is there inter-rater reliability? |46| **Attrition** | Participants drop out non-randomly | How much attrition? Was it differential between groups? |47| **Reverse causation** | The outcome causes the treatment, not vice versa | Is the temporal order established? Could causation run the other way? |48| **P-hacking / multiple comparisons** | Testing many hypotheses and reporting the significant ones | Were hypotheses pre-registered? How many tests were run? |49| **Regression to the mean** | Extreme observations naturally move toward average | Were participants selected for extreme values? Is there a control group? |5051### Step 3 — Assess External Validity5253*Do the findings generalize beyond the specific study?*5455| Dimension | Questions |56|-----------|----------|57| **Population** | Does the sample represent the target population? Were key demographics included? |58| **Setting** | Would results hold in different contexts (lab → field, country A → country B)? |59| **Time** | Would results hold at different times? Are conditions time-dependent? |60| **Treatment fidelity** | In applied research, can the intervention be reproduced elsewhere? |61| **Outcome measures** | Do the measured outcomes map to the outcomes people care about? |6263### Step 4 — Evaluate the Conclusion-Evidence Link6465The most important question: **Do the methods actually justify the specific claims made?**6667Common gaps:68- **Overclaiming**: Methods support association but conclusions claim causation69- **Overgeneralizing**: Sample is specific but conclusions are universal70- **Cherry-picking**: Results are selectively reported (check supplementary materials, pre-registration)71- **Mechanistic overreach**: Results show *that* something happens but claims explain *why*72- **Statistical vs. practical significance**: A statistically significant effect may be too small to matter7374### Step 5 — Assess Replicability Indicators7576Based on known replication predictors, estimate how likely the findings are to replicate:7778| Indicator | Good Sign | Bad Sign |79|-----------|-----------|----------|80| **Sample size** | Large, well-powered | Small, underpowered |81| **Effect size** | Moderate to large | Implausibly large or tiny |82| **Pre-registration** | Hypotheses registered before data collection | No pre-registration; exploratory framed as confirmatory |83| **Transparency** | Data and code available | Proprietary or unavailable |84| **Independent replication** | Results reproduced by other labs | Single lab, never replicated |85| **Methodology** | Standard methods, clear protocol | Novel methods, unclear protocol |86| **Incentive structure** | Null results publishable | Only positive results valued |8788## Output8990```91METHODOLOGY CRITIQUE92────────────────────93Study: [description]94Design: [experimental/quasi-experimental/observational/qualitative/computational/meta-analytic/theoretical]95Claims: [what the study concludes]9697Internal Validity:98 Threats identified:99 - [Threat] — Severity: [high/moderate/low] — Addressed: [yes/partially/no]100 ...101 Overall internal validity: [strong/moderate/weak]102103External Validity:104 Generalizability concerns:105 - [Concern] — Severity: [high/moderate/low]106 ...107 Overall external validity: [strong/moderate/weak]108109Conclusion-Evidence Link:110 [Does the methodology justify the specific claims?]111 Gaps: [overclaiming/overgeneralizing/cherry-picking/none]112113Replicability Assessment: [likely/uncertain/unlikely]114 Key factors: [what drives this assessment]115116Bottom Line:117 [One-paragraph assessment: what we can and cannot conclude from this study]118119Recommended: [e.g., "Look for independent replications", "The association is credible but the causal claim needs experimental evidence"]120```121122## Error Handling123124**Study is outside evaluator's domain expertise:** Apply general methodology principles but flag domain-specific standards the user should consult. Every field has conventions the generalist may not know.125126**User wants a simple thumbs-up/thumbs-down:** Resist. Methodology quality is a spectrum. Provide the assessment with caveats — a flawed study may still provide useful (if uncertain) evidence.127128**The study is well-designed:** Report this as a positive finding. Not every critique finds problems. A methodologically sound study with appropriate claims deserves acknowledgment.
npbuilds