Ab Test Design Brief

A/B Test Design Brief

Use this skill to turn a product change into a decision-ready A/B test brief. It focuses on experiment anatomy: hypothesis, metrics, baselines, variants, eligibility, randomization, confidence, and launch criteria.

Source Traceability

Primary source: Practical A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from chapter 2, especially "Creating a Clear Hypothesis" through "Summarizing the For You A/B Test" in the working text analysis at lines 1203-1942. Related motivation and variant examples come from chapter 1 lines 394-718.

Related Advanced Skills

• experiment-sensitivity-optimization: use when the brief is blocked by MDE, sample size, noisy metrics, CUPED, capping, or too many variants.
• experiment-verification-monitoring: use when the brief needs prelaunch QA, canaries, exposure validation, or active experiment health checks.
• long-term-impact-evaluation: use when the brief needs delayed or sustained impact measurement beyond the initial test window.

Reference Routing

Workflow

1. State the product decision the test must inform.
2. Write a hypothesis with observation, predicted change, audience, and metrics.
3. Choose one primary success metric plus guardrail metrics.
4. Establish the baseline or explain why a proxy metric is being used.
5. Define eligibility, exposure, test variant, and control variant.
6. Choose the randomization unit that preserves a coherent user experience.
7. Record confidence requirements, sample-size assumptions, and launch criteria.

Output Format

# A/B Test Brief

## Decision
[What decision this test will support.]

## Hypothesis
Because [observation], we believe [change] will cause [outcome] for [audience].
We will know this is true when [primary metric] changes without harming [guardrails].

## Metrics
| Metric | Role | Baseline | Target or Concern | Data Source |
|--------|------|----------|-------------------|-------------|

## Variants and Eligibility
- Population:
- Eligibility criteria:
- Exposure event:
- Control:
- Test:
- Randomization unit:

## Confidence Plan
- Minimum detectable effect:
- Sample size or duration:
- Risks to validity:

## Launch Criteria
- Ship if:
- Do not ship if:
- Investigate if:

Quality Bar

• Do not accept a vague "see what happens" experiment.
• Do not let proxy metrics hide missing instrumentation; name the compromise.
• Do not generalize beyond the population that was eligible and exposed.
• Keep variants interpretable: if many things change, the learning becomes weak.