A/B Test Results Readout
Use this skill to turn experiment data into a clear decision. It emphasizes
metric interpretation, data-quality checks, subgroup analysis, ad hoc analysis,
visualization, and launch recommendations.
Source Traceability
Primary source: Practical A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from chapter 4 lines 2950-3742 and chapter 1 lines
572-718. Metric tradeoff context comes from chapter 2 lines 1296-1472.
Related Advanced Skills
trustworthy-experiment-insights: use when the readout needs false positive,
false negative, power, replication, meta-analysis, or suspicious-lift review.
experiment-verification-monitoring: use when result interpretation depends
on whether assignment, exposure, metrics, canaries, or active monitoring were
healthy.
long-term-impact-evaluation: use when short-term readout is not enough to
decide durable product or business impact.
Reference Routing
| Need |
Read |
| Readout concepts |
references/core/knowledge.md |
| Analysis and reporting rules |
references/core/rules.md |
| Example readouts |
references/core/examples.md |
| Step-by-step report workflow |
workflows/prepare-results-readout.md |
Workflow
- Reconstruct the test design: hypothesis, variants, population, and metrics.
- Verify data quality and whether exposure/eligibility match the brief.
- Compare primary and guardrail metrics against baseline and decision rules.
- Run subgroup analysis when averages obscure meaningful differences.
- Investigate outliers, missing data, or surprising movement.
- Visualize results so stakeholders can compare control, test, and segments.
- Recommend ship, stop, iterate, or investigate with caveats.
Output Format
# A/B Test Results Readout
## Executive Decision
[Ship | Stop | Iterate | Investigate] because [reason].
## Test Summary
- Hypothesis:
- Population:
- Control:
- Test:
- Run window:
## Metric Results
| Metric | Role | Control | Test | Change | Interpretation |
|--------|------|---------|------|--------|----------------|
## Segment Findings
| Segment | What changed | Decision impact |
|---------|--------------|-----------------|
## Data Quality Notes
- Eligibility/exposure:
- Missing data:
- Outliers:
- Instrumentation concerns:
## Recommendation
- Decision:
- Rollout conditions:
- Follow-up analysis:
Quality Bar
- Do not hide guardrail regressions behind a primary-metric win.
- Do not overstate subgroup findings; label them exploratory when not
pre-planned.
- Do not show only averages when the product decision depends on user groups.
- Use charts to clarify comparisons, not to decorate the readout.
1---2name: ab-test-results-readout3description: Analyze and communicate A/B test results with metric readouts, subgroup analysis, data-quality checks, ad hoc investigation, visualization, and launch recommendations. Use when interpreting experiment results, preparing an A/B test report, explaining flat or mixed results, checking guardrails, segmenting test/control data, or turning experiment data into a product decision.4license: MIT5compatibility: Codex, Claude Code, and other Agent Skills-compatible clients.6metadata:7 source: https://skillmd.com/skills/hashgraph-online/ab-test-results-readout8 version: "0.1.0"9 displayName: A/B Test Results Readout10 category: Product Management11 tags: practical-ab-testing,ab-testing,experimentation,analytics,data-visualization12---1314# A/B Test Results Readout1516Use this skill to turn experiment data into a clear decision. It emphasizes17metric interpretation, data-quality checks, subgroup analysis, ad hoc analysis,18visualization, and launch recommendations.1920## Source Traceability2122Primary source: Practical A/B Testing by Leemay Nassery. Guidance is23transformed and paraphrased from chapter 4 lines 2950-3742 and chapter 1 lines24572-718. Metric tradeoff context comes from chapter 2 lines 1296-1472.2526## Related Advanced Skills2728- `trustworthy-experiment-insights`: use when the readout needs false positive,29 false negative, power, replication, meta-analysis, or suspicious-lift review.30- `experiment-verification-monitoring`: use when result interpretation depends31 on whether assignment, exposure, metrics, canaries, or active monitoring were32 healthy.33- `long-term-impact-evaluation`: use when short-term readout is not enough to34 decide durable product or business impact.3536## Reference Routing3738| Need | Read |39|------|------|40| Readout concepts | `references/core/knowledge.md` |41| Analysis and reporting rules | `references/core/rules.md` |42| Example readouts | `references/core/examples.md` |43| Step-by-step report workflow | `workflows/prepare-results-readout.md` |4445## Workflow46471. Reconstruct the test design: hypothesis, variants, population, and metrics.482. Verify data quality and whether exposure/eligibility match the brief.493. Compare primary and guardrail metrics against baseline and decision rules.504. Run subgroup analysis when averages obscure meaningful differences.515. Investigate outliers, missing data, or surprising movement.526. Visualize results so stakeholders can compare control, test, and segments.537. Recommend ship, stop, iterate, or investigate with caveats.5455## Output Format5657```markdown58# A/B Test Results Readout5960## Executive Decision61[Ship | Stop | Iterate | Investigate] because [reason].6263## Test Summary64- Hypothesis:65- Population:66- Control:67- Test:68- Run window:6970## Metric Results71| Metric | Role | Control | Test | Change | Interpretation |72|--------|------|---------|------|--------|----------------|7374## Segment Findings75| Segment | What changed | Decision impact |76|---------|--------------|-----------------|7778## Data Quality Notes79- Eligibility/exposure:80- Missing data:81- Outliers:82- Instrumentation concerns:8384## Recommendation85- Decision:86- Rollout conditions:87- Follow-up analysis:88```8990## Quality Bar9192- Do not hide guardrail regressions behind a primary-metric win.93- Do not overstate subgroup findings; label them exploratory when not94 pre-planned.95- Do not show only averages when the product decision depends on user groups.96- Use charts to clarify comparisons, not to decorate the readout.