# Ab Test Design

> Design an a/b test or experiment including hypothesis, metrics, sample size, and analysis plan. Use whenever the user mentions a/b test, experiment, split test, multivariate test, or needs to design a test that produces a decision-quality result.

- Skill: `dragoon0x/ab-test-design` (Agent Skill)
- Install (CLI): `npx skillmds@latest add dragoon0x/ab-test-design`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dragoon0x/ab-test-design/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: dragoon0x (https://skillmd.com/u/dragoon0x)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/dragoon0x/ab-test-design

---


# ab-test-design

most a/b tests don't produce learnings. they produce arguments. a well-designed test produces a decision.

## the elements

### hypothesis

stated in a form that can be falsified:

"if we [change], then [primary metric] will move [direction] by at least [amount] within [timeframe], because [mechanism]."

the `because` matters. it's the reason you'd be confident in the result.

### primary metric

one. the metric that, if it moves, you'd make a decision on. defined precisely.

### secondary metrics

context. helps interpret the primary movement. not the basis for the decision.

### guardrails

things you'd want to know didn't break. if a guardrail crosses a threshold, you'd consider stopping the test even if the primary metric is winning.

### exposure

who sees the variant. user-level random assignment is the default. session-level only if user-level doesn't make sense for the change.

### duration and sample size

calculate before launching. needs:

- expected baseline rate of the primary metric
- minimum detectable effect (the smallest difference that would matter for the decision)
- desired statistical power (80% is typical)
- significance threshold (95% confidence is typical)

if the math says you need 10x your weekly traffic, the test as designed isn't going to give you an answer. redesign the test or the metric.

## the trap of running too short

most tests show winning early because the early adopters of a variant differ from the steady-state population. wait until the planned sample size is reached.

## the trap of running too long

if a test has been running for weeks and isn't conclusive, the effect is probably small enough that the change doesn't matter for the primary metric. accept that and move on.

## the trap of multiple comparisons

testing 4 variants against a control increases the chance of a false positive. either run a single A/B with the strongest variant, or use multiple-comparison corrections in the analysis.

## analysis plan (written before launch)

- what cuts do you plan to look at (by user segment, by platform, etc.)
- what would make you ship the variant
- what would make you not ship
- what would make you run a follow-up test instead of deciding

writing the analysis plan before the data comes in protects against motivated reasoning afterward.

## output

```
hypothesis: [if/then with mechanism]

primary metric: [name, definition, expected baseline, minimum detectable effect, target lift]

secondary metrics: [list with definitions]

guardrails: [list with thresholds that would stop the test]

variants:
  control: [current behavior]
  variant: [the change being tested]

exposure: [user-level / session-level, % of traffic]

sample size: [calculated number, with assumptions]

duration: [planned weeks]

analysis plan:
  cuts to examine: [list]
  decision rule: [ship / no-ship / follow-up criteria]
```