# Ab Test Analysis

> Use when a task needs analysis of A/B test results, interpretation of p-values and confidence intervals, statistical significance checks, or a principled ship/no-ship decision.

- Skill: `jshsakura/ab-test-analysis` (Agent Skill)
- Install (CLI): `npx skillmds@latest add jshsakura/ab-test-analysis`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jshsakura/ab-test-analysis/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: jshsakura (https://skillmd.com/u/jshsakura)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jshsakura/ab-test-analysis

---


## Instructions

Own A/B test analysis as a principled ship/no-ship decision, not a p-value lookup.

Separate statistical significance from practical significance, and catch common analysis traps before recommending action.

Working mode:
1. Establish the experiment design: primary metric, guardrails, pre-specified minimum detectable effect, and duration.
2. Validate test integrity (sample ratio match, sufficient power, no peeking) before trusting any result.
3. Quantify the effect: observed lift, confidence interval, p-value, and whether the result is practically meaningful.
4. Issue a ship / no-ship / iterate / inconclusive verdict with explicit rationale.

Focus on:
- p-value meaning: probability of data this extreme under the null, not the probability the variant is better
- effect size and confidence interval as the decision drivers, not significance alone
- guardrail metrics: never ship a primary win that significantly harms a guardrail
- pre-specified segments only; flag post-hoc segmentation as p-hacking
- common errors: peeking, multiple comparisons, Simpson's paradox, survivorship bias
- frequentist vs bayesian framing and which the tooling actually used
- power and sample adequacy when results are trending but underpowered

Quality checks:
- confirm the verdict aligns with primary metric, effect size, and guardrail status together
- verify no sample ratio mismatch invalidates the test
- check the test ran to its predetermined sample size and duration
- ensure each segment claim was pre-planned, not mined after the fact
- distinguish "no effect detected" from "underpowered, extend the test"

Return:
- results summary table (control vs treatment: n, conversion rate, lift, CI, p-value)
- statistical significance verdict and effect-size interpretation
- guardrail metric status
- ship / no-ship / iterate / inconclusive recommendation with rationale
- next step (ship plan, extension, or learning to capture)

Do not declare a win on statistical significance alone when practical significance or guardrails are unmet unless explicitly requested by the parent agent.