# Ab Testing

> When to activate: A/B testing, split testing, experiment design, statistical significance, sample size, ICE framework, multivariate testing, conversion experiments

- Skill: `mattakushi432/ab-testing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add mattakushi432/ab-testing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/mattakushi432/ab-testing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: Mattakushi432 (https://skillmd.com/u/mattakushi432)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/mattakushi432/ab-testing

---


# A/B Testing

## Experiment Design

### Hypothesis Template
```
We believe that [change] will cause [metric] to [increase/decrease]
because [reasoning based on data/insight].
We'll know this is true when we see [statistical significance at X% confidence].
```

Example:
> "We believe that adding customer logos above the signup CTA will increase trial signups by ≥10% because users cite trust as their #1 objection in exit surveys. We'll know this is true when we see 95% confidence with ≥500 conversions per variant."

### Variable Isolation Rule
Test ONE change at a time per experiment. Multi-variable tests need multivariate setup (MVT) with much larger sample sizes.

### Control vs Treatment
| | Control | Treatment |
|--|---------|-----------|
| Definition | Current experience | Modified experience |
| Traffic split | 50% (typical) | 50% (typical) |
| Changes | None | One specific change |

## Statistical Significance

### Key Concepts
- **Significance level (α)**: Probability of false positive. Standard: α = 0.05 (5%)
- **Confidence**: 1 - α = 95% confidence
- **Power (1-β)**: Probability of detecting a real effect. Target: 80%
- **p-value**: Probability that result is due to chance. Need p < 0.05 to call a winner
- **MDE (Minimum Detectable Effect)**: Smallest improvement worth detecting

### Sample Size Calculator (Python)
```python
import math

def sample_size_per_variant(baseline_cr, mde_relative, alpha=0.05, power=0.80):
    """
    baseline_cr: current conversion rate (e.g. 0.05 for 5%)
    mde_relative: minimum detectable effect as relative change (e.g. 0.10 for 10% lift)
    """
    p1 = baseline_cr
    p2 = baseline_cr * (1 + mde_relative)
    
    z_alpha = 1.96  # for 95% confidence (two-tailed)
    z_beta = 0.842  # for 80% power
    
    p_avg = (p1 + p2) / 2
    n = ((z_alpha * math.sqrt(2 * p_avg * (1 - p_avg)) + 
          z_beta * math.sqrt(p1 * (1-p1) + p2 * (1-p2))) ** 2) / (p2 - p1) ** 2
    
    return math.ceil(n)

# Example: 5% baseline CR, want to detect 10% relative lift
n = sample_size_per_variant(0.05, 0.10)
print(f"Need {n} visitors per variant ({n*2} total)")
# Output: ~3,842 per variant (7,684 total)
```

### Quick Sample Size Reference Table
| Baseline CR | Detect 5% lift | Detect 10% lift | Detect 20% lift |
|------------|---------------|----------------|----------------|
| 2% | ~38,000/variant | ~10,000/variant | ~2,700/variant |
| 5% | ~15,000/variant | ~3,900/variant | ~1,000/variant |
| 10% | ~7,400/variant | ~1,900/variant | ~490/variant |
| 20% | ~3,500/variant | ~900/variant | ~230/variant |

### Duration Calculation
```
Required duration (days) = Sample size per variant / (Daily traffic × traffic_split%)

Example:
  Need: 4,000 per variant
  Daily traffic: 1,000 visitors
  Traffic split: 50%
  Duration = 4,000 / (1,000 × 0.5) = 8 days → round up to 14 days (full 2 weeks)
```

Always run for at least 2 full business cycles (typically 14 days) to account for day-of-week effects.

## Test Prioritization

### ICE Framework
| Criterion | Question | Score 1–10 |
|-----------|----------|-----------|
| **Impact** | If this wins, how much does the metric move? | |
| **Confidence** | How confident are we the change will work? | |
| **Ease** | How easy is it to implement and run? | |

ICE Score = (Impact + Confidence + Ease) / 3

### PIE Framework (alternative)
| Criterion | Definition |
|-----------|-----------|
| **Potential** | How much can this page be improved? (based on analytics) |
| **Importance** | How much traffic / revenue does this page generate? |
| **Ease** | How easy is implementation? |

### Test Backlog Template
| Test | Hypothesis | ICE | Metric | Sample needed | Status |
|------|-----------|-----|--------|--------------|--------|
| Logos above CTA | Trust → more signups | 8.3 | Trial CR | 8,000 | Running |
| Shorter form | Less friction | 7.7 | Lead CR | 15,000 | Queued |
| Video testimonial | Social proof | 6.0 | Demo CR | 4,000 | Backlog |

## Common Mistakes to Avoid

| Mistake | Why It's Wrong | Fix |
|---------|---------------|-----|
| Stopping test early when winning | Peeking problem inflates false positives | Pre-commit to sample size; don't check daily |
| Running overlapping tests | Interaction effects confound results | Serialize tests on same page/flow |
| Testing during unusual periods | Sales events, holidays skew results | Avoid peak seasons unless testing those specifically |
| Low traffic test | Underpowered — can't reach significance | Calculate sample size first |
| Multiple metrics | Higher chance of false positive | Pre-specify ONE primary metric |
| Not segmenting results | Winning average hides losing segments | Always check results by device, source, user segment |

## Multivariate Testing (MVT)

Use only when you need to test COMBINATIONS of changes:

### MVT vs A/B
| | A/B | MVT |
|--|-----|-----|
| Variables | 1 | 2+ simultaneously |
| Variants | 2 | 4–16+ |
| Traffic needed | Low | 10–50× more |
| Insight | Which version wins | Which combination wins AND interaction effects |

### Full Factorial MVT Example
Testing: 2 headlines × 2 images = 4 variants:
- A: Headline 1 + Image 1
- B: Headline 1 + Image 2
- C: Headline 2 + Image 1
- D: Headline 2 + Image 2

## Declaring a Winner

### Decision Criteria
1. Reached pre-specified sample size ✓
2. p-value < 0.05 ✓
3. Ran for minimum 2 business cycles ✓
4. No anomalies in data (traffic spike, tracking issue) ✓
5. Effect consistent across major segments (mobile/desktop, organic/paid) ✓

### When to Ship Without Significance
- Revenue impact is very large (directionally positive, even if not significant)
- Test is directionally consistent across all segments
- Cost of not shipping exceeds cost of being wrong

### Result Documentation
```markdown
## Test: [Name]
- Hypothesis: [stated hypothesis]
- Start: [date] | End: [date]
- Traffic: [N control] / [N treatment]
- Primary metric: [metric]
  - Control: [value] | Treatment: [value] | Δ: [%] | p-value: [p]
- Result: Winner / No significant difference / Inconclusive
- Decision: Ship / Rollback / Extend
- Learnings: [1–3 sentences on what this tells us]
- Next test: [what to test next based on this result]
```