Foundations R Hypothesis Testing And Nonparametrics

R Hypothesis Testing and Nonparametric Methods

When to Use

• Testing a proportion against a fixed value (e.g., "42 of 120 patients responded — does that differ from a 40% historical rate?").
• Comparing paired measurements (before/after, treated/control on the same subject) with n too small or too skewed to trust a paired t-test.
• Comparing two independent groups (e.g., biological replicates, treatment arms) that are non-normal — Mann-Whitney U / Wilcoxon rank-sum instead of t.test.
• Comparing 3+ independent groups nonparametrically (Kruskal-Wallis) and finding which pairs differ (Dunn post-hoc).
• Computing power or required sample size for a binomial test before running an experiment.

Version Compatibility

• R ≥ 4.0, base stats package (ships with R: binom.test, wilcox.test, kruskal.test, p.adjust).
• exactRankTests (CRAN, last updated for R ≥ 3.0) for exact Wilcoxon p-values (wilcox.exact) — needed when ties are present or n is small enough that wilcox.test's exact method refuses.
• PMCMRplus (actively maintained; the older PMCMR is archived on CRAN) for Dunn's post-hoc test after Kruskal-Wallis.

Prerequisites

• install.packages(c("exactRankTests", "PMCMRplus"))
• Familiarity with R's d/p/q/r distribution-function convention (below) and with reading a data.frame.
• Concept prerequisite: understand p-values, null/alternative hypotheses, and one- vs two-sided tests.

R Distribution Function Convention

Suffixes: norm, binom, t, chisq, f, pois, exp, unif, nbinom. lower.tail=FALSE gives P(X > x).

# P(X > 54) for Binomial(100, 0.25)
pbinom(54, size = 100, prob = 0.25, lower.tail = FALSE)

# Normal: P(X > 252) where mean=262.5, sd=12
pnorm(252, mean = 262.5, sd = 12, lower.tail = FALSE)

# Sample size for a 99.5% CI half-width <= 0.5, sigma = 12
ceiling((12 / 0.5 * qnorm(0.995))^2)

Exact Binomial Test + Power

Goal: decide whether an observed proportion differs from a fixed p0, and (before running the study) determine the sample size needed to detect a given effect. Approach: binom.test gives the exact p-value from the binomial CDF; a hand-rolled power function built on qbinom/pbinom lets you scan power across a range of true proportions without extra packages.

# H0: p = 0.5 (e.g. 701 of 1002 patients prefer treatment A). H1: p > 0.5 (right-sided).
binomial_test_report <- function(successes, n, p0, alternative = "two.sided") {
  # Wraps binom.test and also reports the normal-approximation p-value for comparison.
  exact <- binom.test(successes, n, p0, alternative = alternative)
  z <- (successes - n * p0) / sqrt(n * p0 * (1 - p0))
  approx_p <- switch(alternative,
    "greater"   = pnorm(z, lower.tail = FALSE),
    "less"      = pnorm(z, lower.tail = TRUE),
    "two.sided" = 2 * pnorm(-abs(z))
  )
  list(exact_p = exact$p.value, approx_p = approx_p, statistic_z = z)
}

binomial_test_report(701, 1002, 0.5, alternative = "greater")

# Power function for a left-sided test: H0: p=0.3, n=50, alpha=0.05
n <- 50; p_0 <- 0.3
qb <- qbinom(0.05, n, p_0, lower.tail = TRUE)   # critical boundary + 1
power <- function(p) pbinom(qb - 1, n, p, lower.tail = TRUE)
power(0.25)                    # power to detect true p = 0.25
cat("Type II error:", 1 - power(0.25), "\n")

# Required sample size for alpha=0.05, beta=0.025, true p1
required_n <- function(p1, p0 = 0.3) {
  ceiling(((qnorm(0.975) * sqrt(p1 * (1 - p1)) - sqrt(p0 * (1 - p0)) * qnorm(0.05)) / (p0 - p1))^2
}
required_n(0.25)

Sign Test and Wilcoxon Signed-Rank (Paired Data)

Goal: test whether the median of paired differences is zero (e.g., weight before vs. after a diet). Approach: the sign test only counts which member of each pair is larger (equivalent to binom.test(p=0.5)); the Wilcoxon signed-rank test additionally ranks the magnitude of each difference, so it is strictly more powerful when differences are symmetric.

weight_before <- c(89.4, 92.1, 78.3, 101.5, 85.0, 95.2, 88.9, 90.1)
weight_after  <- c(87.1, 90.0, 79.0,  98.2, 83.5, 93.0, 86.2, 88.5)

# Sign test: H1 = weight decreases after diet (left-sided)
b <- sum(weight_after > weight_before)   # count of positive (increase) differences
n <- length(weight_before)
binom.test(b, n, p = 0.5, alternative = "less")

# Wilcoxon signed-rank test — uses magnitude as well as sign
library(exactRankTests)
wilcox.exact(weight_after, weight_before, paired = TRUE, alternative = "less", conf.int = TRUE)

# Manual signed-rank calculation (what wilcox.exact does internally)
manual_signed_rank <- function(x, y) {
  # x, y: paired vectors (e.g. after, before). Returns W+ and W-.
  d <- x - y
  rk <- rank(abs(d))
  signed_rk <- rk * sign(d)
  list(W_plus = sum(signed_rk[signed_rk > 0]), W_minus = sum(-signed_rk[signed_rk < 0]))
}
manual_signed_rank(weight_after, weight_before)

Wilcoxon Rank-Sum / Mann-Whitney U (Independent Samples)

Goal: compare two independent, non-normal (or small-n) samples without assuming equal variance/normality. Approach: wilcox.test(paired=FALSE) (or wilcox.exact for exact p-values with ties); add conf.int=TRUE to get the robust Hodges-Lehmann location estimate alongside the test.

female <- c(118, 122, 130, 115, 128, 121, 119, 124)
male   <- c(135, 128, 140, 132, 138, 130, 142, 136)

# Two-sided Mann-Whitney U / Wilcoxon rank-sum test
wilcox.test(female, male, paired = FALSE, alternative = "two.sided")

# Hodges-Lehmann estimate (median of all pairwise differences) with 95% CI
library(exactRankTests)
wilcox.exact(female, male, paired = FALSE, alternative = "two.sided", conf.int = TRUE)

Kruskal-Wallis Test + Dunn Post-Hoc (3+ Groups)

Goal: nonparametric analogue of one-way ANOVA — test whether ≥3 independent groups share the same distribution, then locate which pairs differ. Approach: kruskal.test on a list of numeric vectors (or formula + data.frame); if significant, run PMCMRplus::kwAllPairsDunnTest with a multiple-testing correction.

soil_type_1 <- c(23.1, 25.4, 22.8, 24.0, 26.1)
soil_type_2 <- c(28.5, 30.2, 27.9, 29.4, 31.0)
soil_type_3 <- c(21.0, 20.5, 22.2, 19.8, 21.5)

kruskal.test(list(soil_type_1, soil_type_2, soil_type_3))

# Dunn's post-hoc test with Benjamini-Hochberg correction
library(PMCMRplus)
crop_df <- data.frame(
  yield = c(soil_type_1, soil_type_2, soil_type_3),
  soil  = factor(rep(c("type1", "type2", "type3"), each = 5))
)
kwAllPairsDunnTest(yield ~ soil, data = crop_df, p.adjust.method = "BH")

Test Selection Guide

Pitfalls

• wilcox.test naming: With two unpaired samples it performs Mann-Whitney U. With paired=TRUE it performs Wilcoxon signed-rank. The name is ambiguous — always specify paired=.
• p.adjust method names: Use "BH" for Benjamini-Hochberg FDR. Passing "fdr" fails silently (not a valid method name).
• t.test/wilcox.test default is two-sided: Use alternative="greater" or alternative="less" only when you have a pre-specified directional hypothesis.
• Exact vs asymptotic Wilcoxon: wilcox.test's built-in exact=TRUE breaks down with ties (falls back to a warning + normal approximation); use exactRankTests::wilcox.exact when ties are present and you still need an exact p-value.
• Kruskal-Wallis ≠ pairwise differences: A significant kruskal.test only means at least one group differs. Follow up with Dunn's test (or pairwise.wilcox.test) plus a correction to find which pairs.
• Hodges-Lehmann is a location estimate, not a mean: It's the median of pairwise averages/differences — robust to outliers (a 10x outlier barely moves it, unlike mean()), but report it, don't confuse it with the sample mean.
• Assumption violations: Check normality (shapiro.test, Q-Q plot) and homoscedasticity before parametric tests; switch to the nonparametric analogue when assumptions fail.

See Also

• foundations-r-regression-correlation-and-diagnostics — parametric follow-up (t-tests, ANOVA, regression assumptions).
• foundations-probability — underlying distribution theory behind d/p/q/r functions.
• bio-experimental-design-power-analysis — general power/sample-size calculations beyond the binomial case.
• bio-experimental-design-multiple-testing — correction methods (p.adjust) for many simultaneous tests.