Mattakushi432 A/B Testing
Experiment Design
Hypothesis Template
We believe that [change] will cause [metric] to [increase/decrease]
because [reasoning based on data/insight].
We'll know this is true when we see [statistical significance at X% confidence].
Example:
"We believe that adding customer logos above the signup CTA will increase trial signups by ≥10% because users cite trust as their #1 objection in exit surveys. We'll know this is true when we see 95% confidence with ≥500 conversions per variant."
Variable Isolation Rule
Test ONE change at a time per experiment. Multi-variable tests need multivariate setup (MVT) with much larger sample sizes.
Control vs Treatment
| Control | Treatment | |
|---|---|---|
| Definition | Current experience | Modified experience |
| Traffic split | 50% (typical) | 50% (typical) |
| Changes | None | One specific change |
Statistical Significance
Key Concepts
- Significance level (α): Probability of false positive. Standard: α = 0.05 (5%)
- Confidence: 1 - α = 95% confidence
- Power (1-β): Probability of detecting a real effect. Target: 80%
- p-value: Probability that result is due to chance. Need p < 0.05 to call a winner
- MDE (Minimum Detectable Effect): Smallest improvement worth detecting
Sample Size Calculator (Python)
import math
def sample_size_per_variant(baseline_cr, mde_relative, alpha=0.05, power=0.80):
"""
baseline_cr: current conversion rate (e.g. 0.05 for 5%)
mde_relative: minimum detectable effect as relative change (e.g. 0.10 for 10% lift)
"""
p1 = baseline_cr
p2 = baseline_cr * (1 + mde_relative)
z_alpha = 1.96 # for 95% confidence (two-tailed)
z_beta = 0.842 # for 80% power
p_avg = (p1 + p2) / 2
n = ((z_alpha * math.sqrt(2 * p_avg * (1 - p_avg)) +
z_beta * math.sqrt(p1 * (1-p1) + p2 * (1-p2))) ** 2) / (p2 - p1) ** 2
return math.ceil(n)
# Example: 5% baseline CR, want to detect 10% relative lift
n = sample_size_per_variant(0.05, 0.10)
print(f"Need {n} visitors per variant ({n*2} total)")
# Output: ~3,842 per variant (7,684 total)
Quick Sample Size Reference Table
| Baseline CR | Detect 5% lift | Detect 10% lift | Detect 20% lift |
|---|---|---|---|
| 2% | ~38,000/variant | ~10,000/variant | ~2,700/variant |
| 5% | ~15,000/variant | ~3,900/variant | ~1,000/variant |
| 10% | ~7,400/variant | ~1,900/variant | ~490/variant |
| 20% | ~3,500/variant | ~900/variant | ~230/variant |
Duration Calculation
Required duration (days) = Sample size per variant / (Daily traffic × traffic_split%)
Example:
Need: 4,000 per variant
Daily traffic: 1,000 visitors
Traffic split: 50%
Duration = 4,000 / (1,000 × 0.5) = 8 days → round up to 14 days (full 2 weeks)
Always run for at least 2 full business cycles (typically 14 days) to account for day-of-week effects.
Test Prioritization
ICE Framework
| Criterion | Question | Score 1–10 |
|---|---|---|
| Impact | If this wins, how much does the metric move? | |
| Confidence | How confident are we the change will work? | |
| Ease | How easy is it to implement and run? |
ICE Score = (Impact + Confidence + Ease) / 3
PIE Framework (alternative)
| Criterion | Definition |
|---|---|
| Potential | How much can this page be improved? (based on analytics) |
| Importance | How much traffic / revenue does this page generate? |
| Ease | How easy is implementation? |
Test Backlog Template
| Test | Hypothesis | ICE | Metric | Sample needed | Status |
|---|---|---|---|---|---|
| Logos above CTA | Trust → more signups | 8.3 | Trial CR | 8,000 | Running |
| Shorter form | Less friction | 7.7 | Lead CR | 15,000 | Queued |
| Video testimonial | Social proof | 6.0 | Demo CR | 4,000 | Backlog |
Common Mistakes to Avoid
| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Stopping test early when winning | Peeking problem inflates false positives | Pre-commit to sample size; don't check daily |
| Running overlapping tests | Interaction effects confound results | Serialize tests on same page/flow |
| Testing during unusual periods | Sales events, holidays skew results | Avoid peak seasons unless testing those specifically |
| Low traffic test | Underpowered — can't reach significance | Calculate sample size first |
| Multiple metrics | Higher chance of false positive | Pre-specify ONE primary metric |
| Not segmenting results | Winning average hides losing segments | Always check results by device, source, user segment |
Multivariate Testing (MVT)
Use only when you need to test COMBINATIONS of changes:
MVT vs A/B
| A/B | MVT | |
|---|---|---|
| Variables | 1 | 2+ simultaneously |
| Variants | 2 | 4–16+ |
| Traffic needed | Low | 10–50× more |
| Insight | Which version wins | Which combination wins AND interaction effects |
Full Factorial MVT Example
Testing: 2 headlines × 2 images = 4 variants:
- A: Headline 1 + Image 1
- B: Headline 1 + Image 2
- C: Headline 2 + Image 1
- D: Headline 2 + Image 2
Declaring a Winner
Decision Criteria
- Reached pre-specified sample size ✓
- p-value < 0.05 ✓
- Ran for minimum 2 business cycles ✓
- No anomalies in data (traffic spike, tracking issue) ✓
- Effect consistent across major segments (mobile/desktop, organic/paid) ✓
When to Ship Without Significance
- Revenue impact is very large (directionally positive, even if not significant)
- Test is directionally consistent across all segments
- Cost of not shipping exceeds cost of being wrong
Result Documentation
## Test: [Name]
- Hypothesis: [stated hypothesis]
- Start: [date] | End: [date]
- Traffic: [N control] / [N treatment]
- Primary metric: [metric]
- Control: [value] | Treatment: [value] | Δ: [%] | p-value: [p]
- Result: Winner / No significant difference / Inconclusive
- Decision: Ship / Rollback / Extend
- Learnings: [1–3 sentences on what this tells us]
- Next test: [what to test next based on this result]