1---2name: ab-test-analysis3description: Rigorous A/B test statistical analysis. Use when analyzing experiment results, calculating statistical significance, checking for sample ratio mismatch, or validating test design before launch.4metadata:5 source: https://skillmd.com/skills/nimrodfisher/ab-test-analysis6---78# A/B Test Analysis910# When to use11- An experiment has finished and the team needs a ship / no-ship recommendation12- Results look directionally positive but the team is unsure if they're statistically significant13- A test has been running for weeks without a clear winner and someone needs to decide whether to continue14- A new experiment needs sample-size planning before launch15- Results are disputed and need a rigorous, documented analysis1617# Process181. **Confirm test design** — verify the hypothesis, the control and treatment definitions, the randomisation unit (user/session/device), the primary metric, any guardrail metrics, and the target split ratio.192. **Check for sample ratio mismatch (SRM)** — run a chi-square test on the actual vs. expected split. If SRM is detected, stop and investigate the randomisation pipeline before interpreting results. Use `scripts/ab_test_analyzer.py --check-srm`.203. **Calculate per-variant metrics** — compute the rate (or mean) and 95% confidence interval for the primary metric in each variant. Document absolute and relative difference.214. **Run the significance test** — execute a two-proportion z-test (for rates) or Welch's t-test (for means). Record z-score, p-value, and 95% CI for the effect. Use `references/statistical_tests_reference.md` if unsure which test applies.225. **Check guardrail metrics** — run the same significance test for each guardrail metric. A significant degradation on any guardrail is a blocker regardless of primary metric results.236. **Produce the recommendation** — synthesise SRM result, power, significance, and guardrail checks into a clear ship / no-ship / extend decision. Quantify the expected business impact if shipped. Record in `assets/ab_test_report_template.md`.2425# Inputs the skill needs26- Test plan or hypothesis document (variant definitions, randomisation unit, primary metric)27- Data with at minimum: user_id, variant assignment, primary metric outcome28- Optional: guardrail metric values per user, daily aggregate data for temporal validity checks29- Target split ratio (e.g., 50/50)30- Minimum detectable effect or business threshold for "worth shipping"3132# Output33- `scripts/ab_test_analyzer.py` — runs SRM check, significance test, power analysis, and guardrail checks from a CSV or summary stats input34- `references/statistical_tests_reference.md` — which test to use and when35- `references/ab_test_design_guide.md` — SRM causes, power planning, peeking and multiple testing36- `assets/ab_test_report_template.md` — structured report: design, results, checks, recommendation, expected impact