Hypothesis Testing Skill
Purpose
Guide appropriate selection and interpretation of statistical hypothesis tests for research data analysis.
Test Selection Decision Tree
Step 1: How many variables?
One variable:
- Categorical → Chi-square goodness of fit
- Continuous → One-sample t-test
Two variables:
- Both categorical → Chi-square test of independence
- One categorical, one continuous → T-test or ANOVA
- Both continuous → Correlation or regression
Three+ variables:
- Multiple predictors → Multiple regression or ANOVA
- Complex designs → Mixed models or advanced methods
Step 2: Check assumptions
For t-tests:
- Independence of observations
- Normality (especially for small N)
- Homogeneity of variance
Violations?
- Non-normal → Mann-Whitney U (non-parametric)
- Unequal variance → Welch's t-test
- Dependent observations → Paired t-test or mixed models
For ANOVA:
- Independence
- Normality
- Homogeneity of variance
- No outliers
Violations?
- Non-normal → Kruskal-Wallis test
- Unequal variance → Welch's ANOVA
- Outliers → Robust methods or transformation
Step 3: Interpret results
Always report:
- Test statistic (t, F, χ²)
- Degrees of freedom
- p-value
- Effect size with CI
- Descriptive statistics
Example:
Independent samples t-test showed a significant difference between
groups, t(98) = 3.45, p < .001, d = 0.69, 95% CI [0.29, 1.09].
The experimental group (M = 45.2, SD = 8.3) scored higher than
control (M = 37.8, SD = 9.1).
Common Tests Reference
| Research Question |
Test |
Assumptions |
| 2 groups, continuous outcome |
Independent t-test |
Normality, equal variance |
| 2 measurements, same people |
Paired t-test |
Normality of differences |
| 3+ groups, one factor |
One-way ANOVA |
Normality, homogeneity |
| 3+ groups, multiple factors |
Factorial ANOVA |
Normality, homogeneity |
| Relationship between variables |
Pearson correlation |
Linearity, normality |
| Predict continuous outcome |
Linear regression |
Linearity, normality of residuals |
| 2 categorical variables |
Chi-square test |
Expected frequencies ≥5 |
| Ordinal data, 2 groups |
Mann-Whitney U |
None (non-parametric) |
| Ordinal data, paired |
Wilcoxon signed-rank |
None (non-parametric) |
Assumption Checking
Normality
Visual: Q-Q plot, histogram
Statistical: Shapiro-Wilk test (N < 50), Kolmogorov-Smirnov (N ≥ 50)
Guideline: Robust to moderate violations if N ≥ 30
Homogeneity of Variance
Visual: Box plots, residual plots
Statistical: Levene's test, Bartlett's test
Guideline: Ratio of largest/smallest variance < 4
Independence
Check: Research design, data collection
Red flags: Time series, clustered data, repeated measures
Solution: Use appropriate model (mixed effects, GEE)
Integration
Use with data-analyst agent for complete statistical analysis workflow and experiment-designer agent for planning appropriate analyses.
Version: 1.0.0
1---2name: hypothesis-test3description: Guide selection and interpretation of statistical hypothesis tests. Use when: (1) Choosing appropriate test for research data, (2) Checking assumptions before analysis, (3) Interpreting test results correctly, (4) Reporting statistical findings, (5) Troubleshooting assumption violations.4---5
6# Hypothesis Testing Skill
7
8## Purpose
9
10Guide appropriate selection and interpretation of statistical hypothesis tests for research data analysis.
11
12## Test Selection Decision Tree
13
14### Step 1: How many variables?
15
16**One variable:**
17- Categorical → Chi-square goodness of fit
18- Continuous → One-sample t-test
19
20**Two variables:**
21- Both categorical → Chi-square test of independence
22- One categorical, one continuous → T-test or ANOVA
23- Both continuous → Correlation or regression
24
25**Three+ variables:**
26- Multiple predictors → Multiple regression or ANOVA
27- Complex designs → Mixed models or advanced methods
28
29### Step 2: Check assumptions
30
31**For t-tests:**
321. Independence of observations
332. Normality (especially for small N)
343. Homogeneity of variance
35
36**Violations?**
37- Non-normal → Mann-Whitney U (non-parametric)
38- Unequal variance → Welch's t-test
39- Dependent observations → Paired t-test or mixed models
40
41**For ANOVA:**
421. Independence
432. Normality
443. Homogeneity of variance
454. No outliers
46
47**Violations?**
48- Non-normal → Kruskal-Wallis test
49- Unequal variance → Welch's ANOVA
50- Outliers → Robust methods or transformation
51
52### Step 3: Interpret results
53
54Always report:
551. **Test statistic** (t, F, χ²)
562. **Degrees of freedom**
573. **p-value**
584. **Effect size with CI**
595. **Descriptive statistics**
60
61**Example:**
62```
63Independent samples t-test showed a significant difference between
64groups, t(98) = 3.45, p < .001, d = 0.69, 95% CI [0.29, 1.09].
65The experimental group (M = 45.2, SD = 8.3) scored higher than
66control (M = 37.8, SD = 9.1).
67```
68
69## Common Tests Reference
70
71| Research Question | Test | Assumptions |
72|------------------|------|-------------|
73| 2 groups, continuous outcome | Independent t-test | Normality, equal variance |
74| 2 measurements, same people | Paired t-test | Normality of differences |
75| 3+ groups, one factor | One-way ANOVA | Normality, homogeneity |
76| 3+ groups, multiple factors | Factorial ANOVA | Normality, homogeneity |
77| Relationship between variables | Pearson correlation | Linearity, normality |
78| Predict continuous outcome | Linear regression | Linearity, normality of residuals |
79| 2 categorical variables | Chi-square test | Expected frequencies ≥5 |
80| Ordinal data, 2 groups | Mann-Whitney U | None (non-parametric) |
81| Ordinal data, paired | Wilcoxon signed-rank | None (non-parametric) |
82
83## Assumption Checking
84
85### Normality
86```
87Visual: Q-Q plot, histogram
88Statistical: Shapiro-Wilk test (N < 50), Kolmogorov-Smirnov (N ≥ 50)
89Guideline: Robust to moderate violations if N ≥ 30
90```
91
92### Homogeneity of Variance
93```
94Visual: Box plots, residual plots
95Statistical: Levene's test, Bartlett's test
96Guideline: Ratio of largest/smallest variance < 4
97```
98
99### Independence
100```
101Check: Research design, data collection
102Red flags: Time series, clustered data, repeated measures
103Solution: Use appropriate model (mixed effects, GEE)
104```
105
106## Integration
107
108Use with data-analyst agent for complete statistical analysis workflow and experiment-designer agent for planning appropriate analyses.
109
110---
111
112**Version:** 1.0.0