# a-b-test-design

> Design rigorous A/B tests with clear hypotheses, controlled variants, appropriate metrics, and sample size calculations.

- Skill: `owl-listener/a-b-test-design` (Agent Skill)
- Install (CLI): `npx skillmds add owl-listener/a-b-test-design`
- Raw SKILL.md: https://api.skillmd.com/api/skills/owl-listener/a-b-test-design/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Product & Planning, Coding & Dev Tools, Data & Analytics, Data Analysis, Product Analytics
- Tags: A B Testing, Experimentation, Hypothesis Testing, Metrics, Sample Size, Statistics
- Author: Owl Listener (https://skillmd.com/u/owl-listener)
- Updated: 2026-07-06
- Page: https://skillmd.com/skills/owl-listener/a-b-test-design

---

# A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
## What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
## Test Structure
### 1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
### 2. Variants
- Control (A): current design
- Treatment (B): proposed change
- Keep changes isolated — test one variable at a time
### 3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
### 4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
### 5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
### 6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.
## Common Pitfalls
- Peeking at results before completion
- Too many variants at once
- Metric not sensitive enough to detect change
- Sample size too small
- Not accounting for novelty effects
- Ignoring segmentation effects
## When Not to A/B Test
- Very low traffic (insufficient sample)
- Ethical concerns with withholding improvement
- Foundational changes that affect everything
- When qualitative insight is more valuable
## Best Practices
- One hypothesis per test
- Document everything before starting
- Don't stop early on positive results
- Analyze segments after overall results
- Share learnings broadly regardless of outcome