# Ab Test Analysis > - Skill: `frank-luongt/ab-test-analysis` (Agent Skill) - Install (CLI): `npx skillmds@latest add frank-luongt/ab-test-analysis` - Raw SKILL.md: https://api.skillmd.com/api/skills/frank-luongt/ab-test-analysis/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: frank-luongt (https://skillmd.com/u/frank-luongt) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/frank-luongt/ab-test-analysis --- --- name: ab-test-analysis description: Analyze A/B test results with statistical rigor — calculate significance, check guardrails, and make ship/extend/stop decisions. Use when evaluating experiment results or interpreting test data. tags: [experimentation, ab-testing, analytics, data-driven] --- # A/B Test Analysis Analyze experiment results with statistical rigor and produce a clear **Ship / Investigate / Extend / Stop** recommendation. This skill complements `ab-test-setup` (which handles experiment design). Use this skill when you have **results to analyze**. --- ## Purpose Most A/B test interpretations are wrong — teams either call tests too early, ignore guardrail metrics, or ship on directional trends without statistical significance. This skill enforces disciplined analysis. --- ## When to Use - An A/B test has completed its planned duration - You have conversion data for control and variant groups - Stakeholders are asking "did the test win?" - You need to decide: ship, extend, or kill ## When NOT to Use - Designing or setting up an experiment (use `ab-test-setup`) - The test hasn't reached minimum sample size yet - You're analyzing observational data (not a controlled experiment) --- ## Required Data (Ask If Missing) | Field | Description | | --- | --- | | **Primary metric** | What the test is trying to improve (e.g., conversion rate) | | **Control group** | Sample size (N) and conversions (C) for the control | | **Variant group** | Sample size (N) and conversions (C) for the variant | | **Test duration** | How long the test ran | | **Planned duration** | How long it was designed to run | | **Guardrail metrics** | Metrics that must not degrade (e.g., revenue, page load time) | | **MDE** | Minimum Detectable Effect used in power calculation | --- ## Analysis Process ### Step 1: Validate the Setup Before analyzing results, check: - [ ] **Sample size adequate?** Compare actual N to planned N from power analysis - [ ] **Duration sufficient?** Must cover at least 1–2 full business cycles (e.g., weekday + weekend) - [ ] **SRM check?** Sample Ratio Mismatch — control and variant should have ~equal N (within 1%). If skewed, the test is invalid. - [ ] **No novelty effects?** If you can, check early vs. late behavior. New UI elements get more clicks initially. **If any check fails, the test results may be unreliable. Flag this before proceeding.** ### Step 2: Calculate Core Statistics For conversion rate tests: ```text Control conversion rate: p_c = C_control / N_control Variant conversion rate: p_v = C_variant / N_variant Relative lift: (p_v - p_c) / p_c × 100% Pooled proportion: p = (C_control + C_variant) / (N_control + N_variant) Standard error: SE = sqrt(p × (1-p) × (1/N_control + 1/N_variant)) Z-score: Z = (p_v - p_c) / SE P-value: two-tailed from Z 95% Confidence Interval: (p_v - p_c) ± 1.96 × SE ``` ### Step 3: Assess Significance | Criterion | Threshold | Status | | --- | --- | --- | | Statistical significance | p-value < 0.05 | Pass / Fail | | Practical significance | Lift > MDE | Pass / Fail | | Confidence interval | Does CI exclude 0? | Pass / Fail | **Both statistical AND practical significance are required to ship.** ### Step 4: Check Guardrail Metrics For each guardrail metric: | Guardrail | Control | Variant | Change | Status | | --- | --- | --- | --- | --- | | [metric name] | [value] | [value] | [+/- %] | OK / Warning / Degraded | A guardrail is **degraded** if it shows a statistically significant negative change. ### Step 5: Make the Decision Use this decision matrix: | Primary Metric | Guardrails | Recommendation | | --- | --- | --- | | Significant positive | All OK | **Ship** — roll out to 100% | | Significant positive | Some degraded | **Investigate** — understand trade-off before deciding | | Not significant, positive trend | All OK | **Extend** — run longer if sample size was insufficient | | Not significant, flat | All OK | **Stop** — no effect detected, free up the experiment slot | | Significant negative | Any | **Don't Ship** — revert and learn from the result | --- ## Output Format ```markdown # A/B Test Results: [Test Name] ## Summary | Field | Value | | --- | --- | | Test name | [name] | | Hypothesis | [We believed X would cause Y] | | Primary metric | [metric name] | | Duration | [start] — [end] ([N] days) | | Decision | **Ship / Investigate / Extend / Stop / Don't Ship** | --- ## Results | Group | Sample Size | Conversions | Rate | | --- | --- | --- | --- | | Control | [N] | [C] | [rate]% | | Variant | [N] | [C] | [rate]% | **Relative lift:** [+/- X.X%] **P-value:** [value] **95% CI:** [[lower]%, [upper]%] **Statistically significant:** Yes / No **Practically significant:** Yes / No (MDE was [X]%) --- ## Guardrail Metrics | Metric | Control | Variant | Change | Status | | --- | --- | --- | --- | --- | | [metric] | [val] | [val] | [change] | OK / Warning | --- ## Recommendation **Decision: [Ship / Investigate / Extend / Stop / Don't Ship]** **Rationale:** [2–3 sentences explaining the decision] **Next steps:** 1. [action] 2. [action] --- ## Learnings - [What we learned from this test, regardless of outcome] - [How this informs future experiments] ``` --- ## Common Pitfalls | Pitfall | Why It's Wrong | Correct Approach | | --- | --- | --- | | Peeking at results daily | Inflates false positive rate | Wait for planned duration and sample size | | Calling it at p=0.06 | "Almost significant" isn't significant | Set the threshold before the test, stick to it | | Ignoring guardrails | Winning on one metric while losing on another | Always check guardrails before shipping | | Post-hoc segmentation | Finding "it worked for mobile users!" after the fact is data mining | Pre-register segments or treat as hypothesis for next test | | Running too many variants | Each variant needs full sample size | Limit to 1–2 variants per test | | Not learning from losses | "It didn't work" is not a learning | Document WHY it didn't work and what to try next | --- ## Anti-Patterns | Avoid | Why | Instead | | --- | --- | --- | | "Directional win" | Not a statistical standard | Require p < 0.05 and lift > MDE | | Shipping without guardrail check | May degrade critical metrics | Always check before shipping | | Ending early because it "looks good" | Sequential testing bias | Run to planned duration | | Not documenting learnings | Same failed experiments get repeated | Maintain an experiment log | --- ## References - Kohavi, R., Tang, D., & Xu, Y. *Trustworthy Online Controlled Experiments* (2020) - [Evan Miller's A/B Test Calculator](https://www.evanmiller.org/ab-testing/) - [Sample Size Calculator](https://www.evanmiller.org/ab-testing/sample-size.html)