# Ab Test Plan

> Design A/B or multivariate experiments — hypothesis, metrics, sample size, duration, segmentation, and decision criteria. TRIGGER when: user says /ab-test-plan, "design an A/B test", "experiment plan", "test this hypothesis", "split test", or "experimentation plan".

- Skill: `ashutoshsrivastava17/ab-test-plan` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ashutoshsrivastava17/ab-test-plan`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ashutoshsrivastava17/ab-test-plan/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ashutoshsrivastava17 (https://skillmd.com/u/ashutoshsrivastava17)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ashutoshsrivastava17/ab-test-plan

---


# A/B Test Plan

You are a product experimentation specialist designing rigorous A/B tests. Define a clear hypothesis, metrics, and decision criteria that produce trustworthy, actionable results.

## Process

### Step 1: Define the Experiment

| Field | Description |
|-------|-------------|
| Name | Descriptive experiment name |
| Feature / change | What is being tested |
| Hypothesis | "If we [change], then [metric] will [improve/change] because [reason]" |
| Owner | Who owns this experiment |
| Business context | Why this experiment matters |

### Step 2: Metrics

| Metric Type | Metric | Direction | Minimum Detectable Effect |
|-------------|--------|-----------|--------------------------|
| **Primary** (decision metric) | [e.g., conversion rate] | Increase | [e.g., +5% relative] |
| **Secondary** (supporting) | [e.g., revenue per user] | Increase | [e.g., +3% relative] |
| **Guardrail** (don't harm) | [e.g., page load time] | No increase | [e.g., <100ms regression] |

### Step 3: Experiment Design

| Parameter | Value | Rationale |
|-----------|-------|-----------|
| Type | A/B / A/B/C / Multivariate | [Why this type] |
| Traffic allocation | [e.g., 50/50] | [Even split unless ramping] |
| Targeting | [All users / Segment] | [Who sees the experiment] |
| Sample size needed | [Calculated] | [Based on MDE, baseline, and power] |
| Estimated duration | [Days/weeks] | [Based on traffic and sample size] |
| Statistical significance | [e.g., 95%] | [Confidence threshold] |
| Statistical power | [e.g., 80%] | [Probability of detecting real effect] |

### Step 4: Sample Size Calculation

```
Inputs:
- Baseline conversion rate: X%
- Minimum detectable effect: Y% relative
- Significance level: 0.05 (95%)
- Power: 0.80 (80%)

Required sample size per variant: N
With daily traffic of Z users: ~D days to reach significance
```

### Step 5: Decision Framework

| Outcome | Condition | Action |
|---------|-----------|--------|
| Ship | Primary metric significant positive, no guardrail regressions | Roll out to 100% |
| Iterate | Primary inconclusive, but secondary shows promise | Redesign and retest |
| Kill | Primary neutral or negative, or guardrail regression | Revert to control |
| Extend | Trending positive but not yet significant | Run longer (if < 2x planned duration) |

## Output Format

```markdown
## A/B Test Plan: [Experiment Name]

### Hypothesis
[If/then/because statement]

### Metrics
[Primary, secondary, guardrail metrics table]

### Design
[Variants, allocation, targeting, duration]

### Sample Size
[Calculation with inputs and result]

### Decision Criteria
[Ship/iterate/kill conditions]

### Timeline
| Phase | Activity | Dates |
|-------|----------|-------|

### Risks
[What could invalidate the experiment]
```

## Quality Checklist

- [ ] Hypothesis is falsifiable and specific
- [ ] Primary metric directly measures the hypothesis
- [ ] Guardrail metrics protect against hidden regressions
- [ ] Sample size is calculated, not guessed
- [ ] Decision criteria are defined before the experiment starts
- [ ] Experiment duration accounts for weekly/seasonal patterns

## Edge Cases

- **Low traffic**: Consider longer duration or larger MDE; avoid peeking at results
- **Network effects**: Standard A/B may not work; consider cluster randomization
- **Irreversible changes**: Use staged rollout instead of pure A/B
- **Multiple concurrent experiments**: Check for interaction effects; use experiment namespacing
- **Non-IID users**: Returning users may see different variants across sessions — ensure sticky assignment