Detect Simpson's Paradox in a Metric
A pooled rate can fall while every subgroup rises, or rise while every subgroup falls, when the mix of groups changes. The aggregate is a weighted average and the weights are doing the talking.
Procedure
- Compute the metric pooled, then per strata for the obvious confounders: platform, channel, geography, account tier.
- Compare the two tables and look for a sign flip between pooled and per-stratum.
- Check whether the mix of strata changed between periods — that shift is the usual cause.
- Recompute by fixing the mix: standardise to a common weight vector and compare like for like:
grp = df.groupby(['segment','period'])['converted'].agg(['sum','count']) rate = grp['sum']/grp['count'] w = df[df.period=='prior'].groupby('segment').size(); w = w/w.sum() print((rate.unstack('period') * w).sum()) # mix-adjusted rate per period - Report both the pooled and the mix-adjusted number, with the stratum breakdown, whenever they disagree.
- Take the unweighted mean of stratum rates only for diagnostics — never present it as the overall rate.
Pitfalls
- A pooled ratio (sum over sum) is not the mean of ratios; averaging per-day rates distorts the total.
- Mobile users quadrupling in share can drag a conversion rate down even as mobile and desktop both improve.
- Small strata with extreme rates swing the unweighted mean; weight by volume.
- Confounders are not only categorical — a continuous variable like account age can hide the same reversal.
- Simpson's paradox is a warning about a lurking variable, not a licence to pick the subgroup with the nicer story.
Verification
python3 - <<'PY'
import pandas as pd d = pd.read_csv('conv.csv') pooled = d.groupby('period').apply(lambda g: g['conv'].sum()/g['n'].sum()) byseg = d.groupby(['segment','period']).apply(lambda g: g['conv'].sum()/g['n'].sum()).unstack() print(pooled.to_string()); print(byseg.to_string()) PY
Report: "Pooled conversion fell 1.1pt while every segment rose; the mix shifted 60% toward low-converting mobile. Mix-adjusted change is +0.4pt — the fall was composition, not performance."