Statistical Tests Selector

Statistical Tests Selector — Statistical Test Selection Guide

A guide for selecting and interpreting appropriate statistical tests based on data type and analysis purpose.

Test Selection Decision Tree

What are you comparing?
├── Mean difference between two groups
│   ├── Independent samples → Normal? → Yes: Independent t-test
│   │                                  → No: Mann-Whitney U
│   └── Paired samples → Normal? → Yes: Paired t-test
│                                 → No: Wilcoxon signed-rank
├── Mean difference among three or more groups
│   ├── Independent → Normal? → Yes: One-way ANOVA → Post-hoc: Tukey HSD
│   │                          → No: Kruskal-Wallis → Post-hoc: Dunn
│   └── Repeated measures → Repeated Measures ANOVA / Friedman
├── Relationship between two variables
│   ├── Continuous × Continuous → Linear? → Yes: Pearson correlation
│   │                                      → No: Spearman rank correlation
│   └── Categorical × Categorical → Chi-squared independence test
├── Proportion difference
│   ├── Two groups → Z-test (proportions)
│   └── Three or more groups → Chi-squared homogeneity test
└── Distribution testing
    ├── Normality → Shapiro-Wilk (n<5000) / K-S test
    └── Homogeneity of variance → Levene's test / Bartlett's test

Core Test Details

Independent Samples t-test

from scipy import stats

# Assumption checks
# 1. Normality
stat, p = stats.shapiro(group_a)
print(f"Normality test: p={p:.4f}")

# 2. Homogeneity of variance
stat, p = stats.levene(group_a, group_b)
print(f"Levene's test: p={p:.4f}")

# Conduct test
if levene_p >= 0.05:
    t, p = stats.ttest_ind(group_a, group_b)  # Equal variance
else:
    t, p = stats.ttest_ind(group_a, group_b, equal_var=False)  # Welch's

# Effect size (Cohen's d)
d = (group_a.mean() - group_b.mean()) / np.sqrt(
    ((len(group_a)-1)*group_a.std()**2 + (len(group_b)-1)*group_b.std()**2)
    / (len(group_a) + len(group_b) - 2)
)

Effect Size Interpretation

ANOVA + Post-hoc Tests

# One-way ANOVA
f_stat, p = stats.f_oneway(group_a, group_b, group_c)

if p < 0.05:
    # Post-hoc test (which groups differ?)
    from statsmodels.stats.multicomp import pairwise_tukeyhsd
    tukey = pairwise_tukeyhsd(
        endog=all_values, groups=all_labels, alpha=0.05
    )
    print(tukey.summary())

Chi-squared Test

# Independence test (categorical × categorical)
contingency = pd.crosstab(df['gender'], df['purchase'])
chi2, p, dof, expected = stats.chi2_contingency(contingency)

# Effect size (Cramer's V)
n = contingency.sum().sum()
v = np.sqrt(chi2 / (n * (min(contingency.shape) - 1)))

Multiple Comparison Correction

from statsmodels.stats.multitest import multipletests

reject, pvals_corrected, _, _ = multipletests(
    p_values, alpha=0.05, method='holm'
)

Power Analysis

from statsmodels.stats.power import TTestIndPower

analysis = TTestIndPower()

# Required sample size calculation
n = analysis.solve_power(
    effect_size=0.5,    # Cohen's d = 0.5 (medium effect)
    alpha=0.05,         # Significance level
    power=0.8,          # 80% power
    alternative='two-sided'
)
print(f"Required sample size per group: {int(np.ceil(n))}")

Correct Interpretation of p-values

When p = 0.03:

✅ Correct interpretation:
"Under the null hypothesis, the probability of observing results this extreme is 3%."

❌ Incorrect interpretations:
"The probability that the alternative hypothesis is true is 97%." (Not Bayesian)
"The effect is large." (Effect size is measured separately)
"The result is important." (Statistical significance ≠ practical importance)

Reporting Template

### Analysis: A/B Group Conversion Rate Comparison

**Hypothesis**: The new design (B) has a higher conversion rate than the original (A)
**Test**: Two-sample proportion z-test (one-tailed)
**Sample**: A: n=1000, conversion 5.2% | B: n=1000, conversion 6.8%
**Result**: z=1.58, p=0.057, 95% CI: [-0.05%, 3.25%]
**Effect Size**: h=0.067 (small)
**Conclusion**: Not statistically significant at the 5% level (p=0.057).
         Power analysis: To achieve 80% power at this effect size,
         n=3,500 per group is needed. Sample expansion recommended.