# Ab Analysis

> Design and read A/B tests: sample size and MDE before the start, validity checks on the split, reading the result with a correction for multiple comparisons, and writing the conclusion for a business audience. Use when asked to "compute the MDE", "how much traffic do we need", "analyse this A/B", "look at the test results", "is it significant", "design an experiment", "why did the groups diverge". Not for offline evaluation on historical data, and not for rolling an arm out to production.

- Skill: `viderstar/ab-analysis` (Agent Skill)
- Install (CLI): `npx skillmds@latest add viderstar/ab-analysis`
- Raw SKILL.md: https://api.skillmd.com/api/skills/viderstar/ab-analysis/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ViderStar (https://skillmd.com/u/viderstar)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/viderstar/ab-analysis

---


# A/B tests

## 1. Before the start - not optional

A test launched without this cannot be interpreted: if no effect is found, you cannot tell
whether there is none or whether there was not enough power.

1. **One primary metric.** Everything else is secondary and cannot be used to declare a win.
2. **MDE and sample size** at alpha = 0.05 and power = 0.8. Compute them from the metric's
   actual variance over the last four weeks, not from a textbook example.
3. **Duration in whole weeks.** Weekdays and weekends behave differently; a partial week
   biases the result.
4. **Name the stopping rule in advance.** "We will look and decide" is peeking, and peeking
   inflates false positives.
5. **Write the hypothesis down before the start**, in the ticket. It is the only protection
   against explaining whatever outcome arrives.

## 2. Check validity before reading the result

- **Sample ratio mismatch.** Observed group sizes against expected, chi-square. A mismatch
  is a defect in the split, and the result must not be read no matter how pretty the
  difference is.
- **A/A on the pre-test period.** If the groups already differed *before* the test, the
  difference is not from the arm.
- **Overlap with other tests** running on the same audience in the same window.
- **Outliers.** In a business with a heavy revenue tail, a single large order moves a
  group's mean visibly. Look at the mean, the median and a winsorised mean together;
  disagreement between them is itself the signal.

## 3. Reading it

- Confidence intervals, not just a p-value. The business needs a range for the effect.
- Several metrics at once means a multiple-comparison correction.
- Ratio metrics (conversion, CTR) need the delta method or a bootstrap, not a plain t-test
  over per-user aggregates.
- **"Not significant" is not "no effect."** It is "the effect is smaller than the MDE".
  Write it that way, and state the MDE.
- Segment slices only count if they were declared in advance. A segment found afterwards
  where "it did work" is a search through noise, not a result.

## 4. Writing it up

Structure: hypothesis → design (metric, MDE, duration, split) → validity checks → result
with confidence intervals → **decision** (ship / do not ship / extend) → what remains
unverified.

Then one separate paragraph in plain language: what the user will see and what you are
doing about it. The business reads that paragraph and nothing else, so it must contain no
p-values and no table names.