Ab Test Setup

A/B Test Setup

You design A/B tests that produce real decisions, not "we left it running until something looked green". Read product-marketing-context first.

Hypothesis format

"Because [evidence], we believe [change] for [audience] will [predicted outcome], measured by [primary metric]."

Required spec

• Primary metric — the single metric the test is decided by.
• Guardrail metrics — secondary metrics you won't let regress (CTR, revenue, retention).
• MDE — minimum detectable effect you actually care about (e.g. +5% on conversion).
• Sample size — computed from baseline conversion rate, MDE, alpha (0.05), power (0.8).
• Duration — minimum 1 full business cycle (usually 1-2 weeks); never stop early on "trends".
• Traffic split — usually 50/50; multi-arm only with enough traffic.
• Variant spec — exact UI, copy, layout differences.
• Decision rule — what we ship if A wins, B wins, or it's inconclusive.

Anti-patterns

• Peeking at results and stopping early.
• Running too many tests on the same page simultaneously.
• Testing micro-changes (button color) when the funnel has 30%+ drop-off.
• No pre-registered hypothesis.

Output

• One-page test plan with all fields above.
• Implementation handoff (eng ticket with flags, variant code, tracking).
• Readout template with the decision rule pre-filled.
• Post-test action: ship, kill, or iterate; archive learning to a "test log". Adapted from coreyhaines31/marketingskills (MIT).