When to use
Use this before shipping any change where you need causal proof it helped — pricing, onboarding, copy, layout, algorithm changes. Use it whenever a wrong decision is expensive or the effect is too small to see by eyeballing a chart.
Not for: prioritizing which idea to test first (use gro-growth-experiment / ICE), no-brainer bug fixes, or changes with too little traffic to ever reach significance — there, ship and monitor, or run a holdout. Also not for measuring long-run retention shifts without a long enough horizon.
Method
- Write a falsifiable hypothesis: "Because [insight], changing [X] will improve [primary metric] for [population]." One primary metric only.
- Choose the primary metric at the decision altitude (e.g. activation rate, not clicks). Pick guardrail metrics that must NOT regress (revenue, latency, refunds, unsubscribes).
- Set the MDE (minimum detectable effect) = the smallest lift worth shipping, from a cost/benefit view — not a guess. Decision point: if the MDE is smaller than plausible reality, the test is underpowered; widen scope or accept a longer run.
- Compute sample size per arm. Two-proportion rule of thumb: n ≈ 16 × p(1−p) / (MDE_absolute)². Use α=0.05, power=0.80. Decision point: if required runtime > 4 weeks, reduce arms, raise MDE, or target a higher-traffic surface.
- Define assignment (unit = user, sticky, hashed), split, and duration covering ≥1 full business cycle (usually 1–2 weeks) to avoid day-of-week bias.
- Pre-register the stop rule: fixed horizon OR a sequential test. Decision point: never peek-and-stop on a fixed-horizon test. Pre-declare how you'll handle a guardrail breach (auto-rollback).
Example
Hypothesis: adding social proof to signup lifts signup completion. Baseline p=20%, MDE=+2pp (to 22%). n ≈ 16 × 0.2 × 0.8 / 0.02² = 16 × 0.16 / 0.0004 = 6,400 per arm → 12,800 total. At 1,000 signups/day that's ~13 days; round to 14 to cover two weekly cycles. Guardrails: paid-conversion and support tickets flat.
Pitfalls
- Tracking many metrics and celebrating whichever wins (multiple comparisons → false positives). One primary, pre-declared.
- Peeking daily and stopping at first significance — inflates false-positive rate massively.
- Running less than a full weekly cycle, so weekday/weekend mix skews the result.
- Ignoring a guardrail regression because the primary metric won.
Output format
Hypothesis: Because <insight>, <change> will improve <primary metric> for <population>.
Primary metric: <metric> | Baseline: <p>
Guardrails: <m1 flat/up>, <m2 flat/up>
MDE: <+X pp / +Y%> (rationale: <cost/benefit>)
Sample size: <n>/arm, <total> total
Assignment: unit=<user>, split=<50/50>, sticky, hashed
Duration: <days> (≥1 business cycle)
Stop rule: <fixed horizon N | sequential>; rollback if <guardrail breach>
Decision rule: ship if primary +≥MDE at p<0.05 AND guardrails intact
1---2name: gro-ab-test-design3description: Design a rigorous A/B test with a falsifiable hypothesis, one primary metric, MDE, sample size, and guardrails.4---56## When to use78Use this before shipping any change where you need *causal proof* it helped — pricing, onboarding, copy, layout, algorithm changes. Use it whenever a wrong decision is expensive or the effect is too small to see by eyeballing a chart.910**Not for:** prioritizing which idea to test first (use gro-growth-experiment / ICE), no-brainer bug fixes, or changes with too little traffic to ever reach significance — there, ship and monitor, or run a holdout. Also not for measuring long-run retention shifts without a long enough horizon.1112## Method13141. Write a falsifiable hypothesis: "Because [insight], changing [X] will improve [primary metric] for [population]." One primary metric only.152. Choose the primary metric at the decision altitude (e.g. activation rate, not clicks). Pick guardrail metrics that must NOT regress (revenue, latency, refunds, unsubscribes).163. Set the MDE (minimum detectable effect) = the smallest lift worth shipping, from a cost/benefit view — not a guess. Decision point: if the MDE is smaller than plausible reality, the test is underpowered; widen scope or accept a longer run.174. Compute sample size per arm. Two-proportion rule of thumb: n ≈ 16 × p(1−p) / (MDE_absolute)². Use α=0.05, power=0.80. Decision point: if required runtime > 4 weeks, reduce arms, raise MDE, or target a higher-traffic surface.185. Define assignment (unit = user, sticky, hashed), split, and duration covering ≥1 full business cycle (usually 1–2 weeks) to avoid day-of-week bias.196. Pre-register the stop rule: fixed horizon OR a sequential test. Decision point: never peek-and-stop on a fixed-horizon test. Pre-declare how you'll handle a guardrail breach (auto-rollback).2021## Example2223Hypothesis: adding social proof to signup lifts signup completion. Baseline p=20%, MDE=+2pp (to 22%). n ≈ 16 × 0.2 × 0.8 / 0.02² = 16 × 0.16 / 0.0004 = 6,400 per arm → 12,800 total. At 1,000 signups/day that's ~13 days; round to 14 to cover two weekly cycles. Guardrails: paid-conversion and support tickets flat.2425## Pitfalls2627- Tracking many metrics and celebrating whichever wins (multiple comparisons → false positives). One primary, pre-declared.28- Peeking daily and stopping at first significance — inflates false-positive rate massively.29- Running less than a full weekly cycle, so weekday/weekend mix skews the result.30- Ignoring a guardrail regression because the primary metric won.3132## Output format3334```35Hypothesis: Because <insight>, <change> will improve <primary metric> for <population>.36Primary metric: <metric> | Baseline: <p>37Guardrails: <m1 flat/up>, <m2 flat/up>38MDE: <+X pp / +Y%> (rationale: <cost/benefit>)39Sample size: <n>/arm, <total> total40Assignment: unit=<user>, split=<50/50>, sticky, hashed41Duration: <days> (≥1 business cycle)42Stop rule: <fixed horizon N | sequential>; rollback if <guardrail breach>43Decision rule: ship if primary +≥MDE at p<0.05 AND guardrails intact44```
Matrixx0070