Experiment Stats

Experiment Stats

Compute real inferential statistics for A/B and multivariate marketing experiments, then apply a dual promote-gate to decide whether a variant actually won. The p-value and confidence interval behind that call come from bootstrap resampling and a hand-implemented Mann-Whitney rank-sum test, run directly against the outcome data you provide.

What it does

Given a control group and one or more variant groups of outcome data, scripts/ab_stats.py:

1. Computes lift: (variant_mean - control_mean) / control_mean.
2. Computes a bootstrap confidence interval on that lift (seeded resampling, default 10,000 iterations).
3. Runs a significance test:
   • metric: "conversion" (binary 0/1 outcomes) -> two-proportion z-test.
   • metric: "value" (continuous outcomes) -> Mann-Whitney U test, with the rank-sum computed by hand (tie correction + continuity correction).
4. Applies the dual promote-gate and returns PROMOTE, HOLD, or REJECT with the numbers behind the call.

When to use

Invoke this any time a promote/hold/reject call needs to rest on a computed p-value and a real lift interval. Typical callers:

• The closed-loop-marketing discipline, at the point where it decides whether to promote a winning variant to full rollout.
• Any A/B or multivariate test report on marketing copy, pricing, onboarding flows, or ad creative that ends in a promote/hold/reject call.
• Any experiment dashboard that flags a result "significant" without also checking whether the lift clears the bar for shipping it.
• Any place a lift number is about to be quoted without the confidence interval or p-value behind it.

How to run it

echo '{
  "control": [0, 1, 0, 0, 1, 0, 0, 0, 1, 0],
  "variants": {"B": [1, 1, 0, 1, 0, 1, 1, 0, 1, 0]},
  "metric": "conversion",
  "alpha": 0.05,
  "min_lift": 0.15
}' | python3 scripts/ab_stats.py

Or from a file: python3 scripts/ab_stats.py experiment.json.

Input JSON fields:

Output is JSON, one result block per variant, each with lift, bootstrap_ci, test (which test ran), z, p_value, decision, and a human-readable reason.

How the promote-gate decision works

A variant is promoted only when both conditions hold:

• p_value < alpha (statistically significant), and
• lift >= min_lift (the effect is large enough to matter).

That last row is what the min_lift floor exists for: at large enough sample sizes, a trivial lift can clear p < 0.05 on significance alone, and the floor stops that from reading as a win. See scripts/test_ab_stats.py case (d) for a worked example.

Validation and tests

Run python3 scripts/test_ab_stats.py from this directory. It runs each fixture below, prints PASS or FAIL for every assertion, and exits 0 only if all of them pass.

• a hand-verifiable Mann-Whitney U case (no ties, fully separated samples)
• a clear-winner case that must PROMOTE
• a no-difference case (identical distributions) that must HOLD
• a large-sample, small-lift case that is statistically significant but must still HOLD -- proving the dual gate does what it's for

Caveats

• The Mann-Whitney normal approximation is asymptotic; on very small samples (roughly n < 10 per group) it will disagree somewhat with the exact permutation p-value. It gets closer as sample size grows, and the tie/continuity corrections keep it well-behaved in the meantime.
• ab_stats.py fails with a non-zero exit and a message on stderr for bad input (missing fields, wrong types, empty arrays, non-binary values under metric: "conversion") -- it never silently substitutes a default or drops bad rows.

Attribution

The idea of pairing a bootstrap lift interval with a promote-gate for marketing experiments is drawn from the growth-engine skill in ericosiu/ai-marketing-skills (MIT licensed). This implementation is an independent clean-room reimplementation in pure Python: the statistical routines (rank-sum Mann-Whitney with tie/continuity correction, two-proportion z-test, the bootstrap resampler, and the dual-gate logic) were written from scratch for this skill and share no code with the original repository.