Hypothesis Testing
Framework
IRON LAW: Statistical Significance ≠ Practical Significance
A p-value < 0.05 means the result is unlikely under the null hypothesis.
It does NOT mean the result is important, large, or practically meaningful.
With a large enough sample, a 0.1% conversion rate difference becomes
"statistically significant" but is practically worthless.
ALWAYS report effect size alongside p-value.
IRON LAW: State Hypotheses BEFORE Looking at Data
H₀ (null) and H₁ (alternative) must be defined before data analysis.
Choosing hypotheses after seeing the data = p-hacking = scientific fraud.
"We found an interesting pattern, let's test it on the same data" is invalid.
Core Concepts
| Concept |
Definition |
| H₀ (Null) |
Default assumption: no effect, no difference |
| H₁ (Alternative) |
What you want to show: there IS an effect/difference |
| p-value |
Probability of seeing this result (or more extreme) IF H₀ is true |
| α (significance level) |
Threshold for rejecting H₀ (typically 0.05) |
| Type I error (α) |
Rejecting H₀ when it's actually true (false positive) |
| Type II error (β) |
Failing to reject H₀ when H₁ is true (false negative) |
| Power (1-β) |
Probability of detecting a real effect (target: ≥ 0.8) |
| Effect size |
Magnitude of the difference (Cohen's d, odds ratio, R²) |
Test Selection Guide
| Data Type |
Groups |
Test |
| Continuous, normal, 2 groups |
Independent |
Independent t-test |
| Continuous, normal, 2 groups |
Paired/before-after |
Paired t-test |
| Continuous, normal, 3+ groups |
Independent |
One-way ANOVA |
| Continuous, non-normal |
2 groups |
Mann-Whitney U |
| Categorical |
2+ groups |
Chi-square test |
| Continuous, relationship |
2 variables |
Pearson correlation (normal) / Spearman (non-normal) |
| Binary outcome |
Predictors |
Logistic regression |
Testing Process
- State hypotheses: H₀ and H₁ with specific parameters
- Choose test: Based on data type, distribution, and groups (use guide above)
- Set α: Usually 0.05 (justify if different)
- Calculate: Run the test, get test statistic and p-value
- Decide: p < α → reject H₀; p ≥ α → fail to reject H₀
- Report: Effect size + confidence interval + p-value (not just "significant")
Output Format
# Hypothesis Test: {Research Question}
## Hypotheses
- H₀: {null — no effect/difference}
- H₁: {alternative — there IS an effect/difference}
- α = {0.05 or other}
## Test Selection
- Test: {name}
- Rationale: {why this test fits the data}
- Assumptions checked: {normality, independence, equal variance}
## Results
- Test statistic: {value}
- p-value: {value}
- Effect size: {value and interpretation}
- 95% CI: [{lower}, {upper}]
## Decision
{Reject / Fail to reject H₀}
## Interpretation
{What this means in practical terms, with effect size context}
Gotchas
- "Fail to reject H₀" ≠ "H₀ is true": Absence of evidence is not evidence of absence. You may lack power to detect a real effect.
- Multiple comparisons inflate Type I error: Testing 20 hypotheses at α=0.05 → expect 1 false positive by chance. Apply Bonferroni or FDR correction.
- Check assumptions before testing: t-test assumes normality and equal variance. Violating assumptions invalidates results. Use non-parametric alternatives when assumptions fail.
- Sample size determines power: Small samples miss real effects (Type II error). Calculate required sample size BEFORE collecting data.
- p-value is NOT the probability that H₀ is true: It's the probability of the data given H₀. These are fundamentally different things (base rate fallacy).
References
- For sample size calculation, see
references/sample-size.md
- For non-parametric test alternatives, see
references/nonparametric-tests.md
1---2name: stat-hypothesis-testing3description: "Conduct statistical hypothesis testing including null/alternative hypothesis formulation, p-values, Type I/II errors, and test statistic selection. Use this skill when the user needs to determine whether a result is statistically significant, choose the right statistical test, interpret p-values correctly, or evaluate research findings — even if they say 'is this result significant', 'which statistical test should I use', or 'what does this p-value mean'.".4---56# Hypothesis Testing78## Framework910```11IRON LAW: Statistical Significance ≠ Practical Significance1213A p-value < 0.05 means the result is unlikely under the null hypothesis.14It does NOT mean the result is important, large, or practically meaningful.15With a large enough sample, a 0.1% conversion rate difference becomes16"statistically significant" but is practically worthless.1718ALWAYS report effect size alongside p-value.19```2021```22IRON LAW: State Hypotheses BEFORE Looking at Data2324H₀ (null) and H₁ (alternative) must be defined before data analysis.25Choosing hypotheses after seeing the data = p-hacking = scientific fraud.26"We found an interesting pattern, let's test it on the same data" is invalid.27```2829### Core Concepts3031| Concept | Definition |32|---------|-----------|33| **H₀ (Null)** | Default assumption: no effect, no difference |34| **H₁ (Alternative)** | What you want to show: there IS an effect/difference |35| **p-value** | Probability of seeing this result (or more extreme) IF H₀ is true |36| **α (significance level)** | Threshold for rejecting H₀ (typically 0.05) |37| **Type I error (α)** | Rejecting H₀ when it's actually true (false positive) |38| **Type II error (β)** | Failing to reject H₀ when H₁ is true (false negative) |39| **Power (1-β)** | Probability of detecting a real effect (target: ≥ 0.8) |40| **Effect size** | Magnitude of the difference (Cohen's d, odds ratio, R²) |4142### Test Selection Guide4344| Data Type | Groups | Test |45|-----------|--------|------|46| Continuous, normal, 2 groups | Independent | Independent t-test |47| Continuous, normal, 2 groups | Paired/before-after | Paired t-test |48| Continuous, normal, 3+ groups | Independent | One-way ANOVA |49| Continuous, non-normal | 2 groups | Mann-Whitney U |50| Categorical | 2+ groups | Chi-square test |51| Continuous, relationship | 2 variables | Pearson correlation (normal) / Spearman (non-normal) |52| Binary outcome | Predictors | Logistic regression |5354### Testing Process55561. **State hypotheses**: H₀ and H₁ with specific parameters572. **Choose test**: Based on data type, distribution, and groups (use guide above)583. **Set α**: Usually 0.05 (justify if different)594. **Calculate**: Run the test, get test statistic and p-value605. **Decide**: p < α → reject H₀; p ≥ α → fail to reject H₀616. **Report**: Effect size + confidence interval + p-value (not just "significant")6263## Output Format6465```markdown66# Hypothesis Test: {Research Question}6768## Hypotheses69- H₀: {null — no effect/difference}70- H₁: {alternative — there IS an effect/difference}71- α = {0.05 or other}7273## Test Selection74- Test: {name}75- Rationale: {why this test fits the data}76- Assumptions checked: {normality, independence, equal variance}7778## Results79- Test statistic: {value}80- p-value: {value}81- Effect size: {value and interpretation}82- 95% CI: [{lower}, {upper}]8384## Decision85{Reject / Fail to reject H₀}8687## Interpretation88{What this means in practical terms, with effect size context}89```9091## Gotchas9293- **"Fail to reject H₀" ≠ "H₀ is true"**: Absence of evidence is not evidence of absence. You may lack power to detect a real effect.94- **Multiple comparisons inflate Type I error**: Testing 20 hypotheses at α=0.05 → expect 1 false positive by chance. Apply Bonferroni or FDR correction.95- **Check assumptions before testing**: t-test assumes normality and equal variance. Violating assumptions invalidates results. Use non-parametric alternatives when assumptions fail.96- **Sample size determines power**: Small samples miss real effects (Type II error). Calculate required sample size BEFORE collecting data.97- **p-value is NOT the probability that H₀ is true**: It's the probability of the data given H₀. These are fundamentally different things (base rate fallacy).9899## References100101- For sample size calculation, see `references/sample-size.md`102- For non-parametric test alternatives, see `references/nonparametric-tests.md`
charlieviettq