Mohit Aggarwal A/B Test Planner Skill
Design experiments that produce trustworthy results — not just directional signals. Every test output includes hypothesis, success metrics, sample size, duration, and a results interpretation guide.
Required Inputs
Ask the user for these if not provided:
- What is being tested (feature, UI change, copy, pricing, onboarding step)
- Hypothesis (or ask to help formulate one)
- Primary metric (conversion rate, click-through, completion rate, etc.)
- Baseline rate and minimum detectable effect (MDE)
- Daily eligible users (to calculate duration)
Experiment Design Checklist
Before running any test, confirm:
- Clear hypothesis with predicted direction
- Single primary metric (plus up to 2 guardrail metrics)
- Minimum detectable effect (MDE) defined
- Sample size calculated
- Test duration estimated
- Segment isolated (no overlap with other running tests)
- Rollback plan defined
Hypothesis Template
"We believe that [change] will cause [primary metric] to [increase/decrease] by [X%] for [user segment], because [rationale based on data or insight]."
Never run a test without a directional hypothesis. "Let's just see what happens" is not a hypothesis.
Sample Size Calculator Logic
Use this formula (provide the output, not the formula, to the user):
- Baseline conversion rate: Current rate of primary metric
- MDE: Smallest change worth detecting (recommend 10–20% relative lift for most features)
- Statistical power: 80% (standard)
- Significance level: 95% (p < 0.05)
For common scenarios, provide pre-calculated estimates:
| Baseline Rate | MDE (Relative) | Required Sample per Variant |
|---|---|---|
| 5% | 20% | ~19,000 |
| 10% | 15% | ~14,000 |
| 20% | 10% | ~15,000 |
| 40% | 10% | ~9,500 |
| 60% | 5% | ~42,000 |
Always warn: "These are estimates. Use a tool like Evan Miller's calculator or Statsig for precision."
Test Duration Guidance
Minimum: 2 full weeks (to capture weekly seasonality) Maximum: 4 weeks (novelty effect distorts results beyond this)
Duration = Required sample ÷ (Daily traffic × % exposed)
Flag if traffic is too low to reach significance in under 8 weeks — recommend a different approach (e.g., holdout test, qualitative research).
Output Format
A/B Test Plan — [Test Name] — [Date]
Hypothesis:
[Filled hypothesis template]
Variants:
- Control (A): [Current experience]
- Treatment (B): [Changed experience — be specific]
Primary Metric: [Metric name + how measured] Guardrail Metrics: [Metrics that must not degrade]
Target Segment: [Who sees the test — % of traffic, user type] Traffic Split: [50/50 recommended unless ramp-up needed]
Sample Size Required: ~[N] users per variant Estimated Duration: [X] weeks (based on [Y] daily eligible users) Significance Threshold: 95% confidence, 80% power
Exclusions: [Any user segments to exclude and why]
Rollback Trigger: If [guardrail metric] degrades by [X%], stop the test immediately.
Results Interpretation Guide:
- ✅ Ship if: Treatment shows [X%]+ lift on primary metric at 95% confidence AND guardrail metrics are stable
- 🔄 Iterate if: Direction is positive but not significant — consider extending or redesigning
- ❌ Reject if: No lift or negative direction at significance
- ⚠️ Inconclusive: Do not ship. Do not call it a win.
Guidelines
- Always recommend against peeking at results before the test reaches planned sample size — explain p-hacking risk
- If user wants to test multiple variants, explain the multiple comparisons problem and recommend a Bonferroni correction or a Bayesian approach
- If traffic is very low (<1,000 users/day), recommend qualitative alternatives: moderated testing, 5-second tests, or user interviews
- Never approve a test with no guardrail metrics — always protect revenue, retention, or core engagement
Anti-Patterns
- Do not run a test without a directional hypothesis — "let's see what happens" produces uninterpretable results
- Do not declare a winner before reaching the pre-planned sample size — peeking at results inflates false positive rates
- Do not test multiple independent changes in a single variant — you won't know which change caused the result
- Do not use engagement metrics (clicks, time-on-page) as the primary metric when the goal is revenue or retention — proxy metrics mislead
- Do not ignore guardrail metrics — a conversion lift that causes a support ticket spike is not a win
Deeper Materials
This skill ships with support files — use them when they are available:
references/test-validity-traps.md— The Validity Traps That Quietly Invalidate A/B Tests. Apply it while producing the output; it carries the calibration and judgment calls the method summary above compresses.templates/test-plan.md— a fill-in version of the deliverable with the quality gates inline. Offer it when the user wants to work the document themselves rather than have it generated.
Scoring Rubric (0–40)
Score any output of this skill before handing it over; 32+ is ship-quality.
| Dimension | 0 | 5 | 10 |
|---|---|---|---|
| Statistical rigour | No sample size, or a number with no stated baseline/MDE behind it | Sample size present but MDE is guessed or copied from the lookup table without checking the actual baseline; power/significance unstated | Sample size derived from the stated baseline and MDE at 80% power / 95% confidence, duration checked against real daily traffic and the 2–4 week window, and the low-traffic escape hatch invoked if it doesn't fit |
| Hypothesis discipline | "Let's see what happens" — no direction, no magnitude, or multiple changes bundled into one variant | Directional hypothesis but missing magnitude, segment, or the evidence-based because; variant purity not confirmed | Full template filled (change, metric, direction, magnitude, segment, rationale citing data), and the treatment isolates exactly one change with excluded ideas named as follow-up tests |
| Guardrails & rollback | No guardrail metrics, or a rollback line with no threshold | Guardrails named but denominators/definitions ambiguous; rollback trigger vague ("if things look bad") | 1–2 guardrails protecting revenue or core engagement with pre-agreed definitions, concrete rollback thresholds, and the peeking-vs-harm-monitoring distinction handled explicitly |
| Decision readiness | No interpretation guide; results will be argued about after the fact | Ship/iterate/reject listed but thresholds fuzzy; inconclusive outcome missing or treated as a soft win | All four outcomes (ship / iterate / reject / inconclusive) mapped to pre-committed thresholds, including what an inconclusive result costs and what each outcome changes next |
Quality Checks
- Hypothesis is directional (predicts a specific direction and magnitude, not "let's see")
- Primary metric is singular (guardrail metrics are secondary)
- Sample size is calculated from actual MDE and baseline (not guessed)
- Test duration accounts for weekly seasonality (minimum 2 weeks)
- Guardrail metrics are defined (at least one to protect revenue or core engagement)
- Rollback trigger is specified with a concrete threshold