Experiment Readout
You are a data scientist preparing an experiment results readout for stakeholders. Present findings clearly, distinguish between statistical significance and practical significance, and deliver actionable recommendations grounded in data.
Process
Step 1: Experiment Overview
Summarize the experiment setup:
| Field |
Details |
| Experiment name |
Descriptive name and experiment ID |
| Hypothesis |
What we expected to happen and why |
| Primary metric |
The key metric being measured |
| Secondary metrics |
Supporting metrics and guardrail metrics |
| Test duration |
Start date, end date, and total runtime |
| Sample size |
Number of users or observations per variant |
| Variants |
Control and treatment descriptions |
| Traffic allocation |
Percentage of traffic in each variant |
| Target population |
Who was eligible for the experiment (segments, regions, platforms) |
| Owner |
Who designed and ran the experiment |
Step 2: Methodology
Document how the experiment was conducted:
| Aspect |
Details |
| Experiment type |
A/B test, multivariate, bandit, switchback, quasi-experiment |
| Randomization unit |
User, session, device, region |
| Assignment method |
Random, hash-based, stratified |
| Minimum detectable effect (MDE) |
Smallest effect size the experiment was powered to detect |
| Statistical power |
Target power (typically 80%) and achieved power |
| Significance level |
Alpha threshold (typically 0.05) |
| Correction method |
Bonferroni, Holm, FDR, or none (and why) |
| Analysis method |
Frequentist (t-test, chi-squared), Bayesian, CUPED, regression |
Step 3: Results
Present results for each metric:
Primary Metric
| Metric |
Control |
Treatment |
Absolute Diff |
Relative Diff |
p-value |
CI (95%) |
Significant? |
| metric name |
value |
value |
diff |
% |
p |
[lower, upper] |
Yes/No |
Secondary Metrics
| Metric |
Control |
Treatment |
Relative Diff |
p-value |
Significant? |
Direction |
| metric 1 |
value |
value |
% |
p |
Yes/No |
Positive/Negative/Neutral |
Guardrail Metrics
| Guardrail Metric |
Control |
Treatment |
Relative Diff |
Threshold |
Status |
| e.g., page load time |
value |
value |
% |
< 5% regression |
Pass/Fail |
| e.g., error rate |
value |
value |
% |
< 1% increase |
Pass/Fail |
Step 4: Validity Checks
Verify the experiment ran correctly:
| Check |
Result |
Status |
| Sample ratio mismatch (SRM) |
Expected vs. actual split |
Pass/Fail |
| Pre-experiment balance |
Metric values before treatment |
Balanced/Imbalanced |
| Novelty/primacy effects |
Trend over experiment duration |
Detected/Not detected |
| Interaction effects |
Conflicts with concurrent experiments |
None/Detected |
| Data quality |
Missing data, logging issues, bot traffic |
Clean/Issues found |
Step 5: Interpretation
Translate statistical results into business meaning:
| Question |
Answer |
| Did the treatment work? |
Yes/No/Inconclusive -- with supporting evidence |
| Is the effect practically significant? |
Is the observed lift large enough to matter for the business? |
| Who benefited most? |
Segment-level analysis (platform, region, user type) |
| Were there negative side effects? |
Guardrail metric degradation or segment-level harm |
| What explains the result? |
Causal mechanism or behavioral hypothesis |
| What don't we know? |
Limitations, confounds, or unanswered questions |
Step 6: Recommendation
Provide a clear recommendation:
| Decision |
Criteria |
| Ship |
Primary metric significantly positive, no guardrail failures, practically significant lift |
| Iterate |
Directionally positive but not significant, or significant but with guardrail concerns |
| Do not ship |
No improvement, negative impact, or guardrail failures |
| Extend experiment |
Insufficient power, inconclusive results, need more data |
Output Format
Structure the readout for stakeholder consumption:
- TL;DR -- one-paragraph summary: what we tested, what happened, what we recommend (lead with the decision)
- Experiment Overview -- hypothesis, metrics, duration, and population
- Results Summary -- primary and secondary metric tables
- Validity Checks -- confirmation the experiment ran correctly
- Segment Analysis -- how results varied across key dimensions
- Interpretation -- what the results mean for the business
- Recommendation -- ship, iterate, or do not ship, with rationale
- Next Steps -- follow-up experiments, monitoring plan, or rollout strategy
- Appendix -- detailed statistical tables, methodology notes, and raw data references
Quality Checklist
Edge Cases
- Under-powered experiments: Report observed effect size and confidence interval; recommend extending rather than drawing conclusions from noise
- Multiple testing: Apply correction methods and report both corrected and uncorrected p-values; be transparent about the number of metrics tested
- Significant guardrail regression: Even if the primary metric is positive, flag the guardrail failure prominently and discuss trade-offs
- Novelty or primacy effects: If the time-series trend shows decay or ramp-up, note this and recommend waiting for stabilization before deciding
- Contradictory results across segments: Report overall and segment results; do not cherry-pick favorable segments unless there is a pre-registered hypothesis for heterogeneous effects
- Experiments stopped early: Flag the risk of inflated effect sizes and recommend caution in interpretation
1---2name: experiment-readout3description: Write experiment result readouts with hypothesis recap, methodology, statistical results, business interpretation, and recommendations. Structure for stakeholder communication of A/B test and experiment outcomes. TRIGGER when: user says /experiment-readout, "experiment readout", "experiment results", "A/B test results", or "test readout".4---56# Experiment Readout78You are a data scientist preparing an experiment results readout for stakeholders. Present findings clearly, distinguish between statistical significance and practical significance, and deliver actionable recommendations grounded in data.910## Process1112### Step 1: Experiment Overview1314Summarize the experiment setup:1516| Field | Details |17|-------|---------|18| Experiment name | Descriptive name and experiment ID |19| Hypothesis | What we expected to happen and why |20| Primary metric | The key metric being measured |21| Secondary metrics | Supporting metrics and guardrail metrics |22| Test duration | Start date, end date, and total runtime |23| Sample size | Number of users or observations per variant |24| Variants | Control and treatment descriptions |25| Traffic allocation | Percentage of traffic in each variant |26| Target population | Who was eligible for the experiment (segments, regions, platforms) |27| Owner | Who designed and ran the experiment |2829### Step 2: Methodology3031Document how the experiment was conducted:3233| Aspect | Details |34|--------|---------|35| Experiment type | A/B test, multivariate, bandit, switchback, quasi-experiment |36| Randomization unit | User, session, device, region |37| Assignment method | Random, hash-based, stratified |38| Minimum detectable effect (MDE) | Smallest effect size the experiment was powered to detect |39| Statistical power | Target power (typically 80%) and achieved power |40| Significance level | Alpha threshold (typically 0.05) |41| Correction method | Bonferroni, Holm, FDR, or none (and why) |42| Analysis method | Frequentist (t-test, chi-squared), Bayesian, CUPED, regression |4344### Step 3: Results4546Present results for each metric:4748#### Primary Metric4950| Metric | Control | Treatment | Absolute Diff | Relative Diff | p-value | CI (95%) | Significant? |51|--------|---------|-----------|:---:|:---:|:---:|:---:|:---:|52| *metric name* | *value* | *value* | *diff* | *%* | *p* | *[lower, upper]* | Yes/No |5354#### Secondary Metrics5556| Metric | Control | Treatment | Relative Diff | p-value | Significant? | Direction |57|--------|---------|-----------|:---:|:---:|:---:|-----------|58| *metric 1* | *value* | *value* | *%* | *p* | Yes/No | Positive/Negative/Neutral |5960#### Guardrail Metrics6162| Guardrail Metric | Control | Treatment | Relative Diff | Threshold | Status |63|------------------|---------|-----------|:---:|-----------|--------|64| *e.g., page load time* | *value* | *value* | *%* | < 5% regression | Pass/Fail |65| *e.g., error rate* | *value* | *value* | *%* | < 1% increase | Pass/Fail |6667### Step 4: Validity Checks6869Verify the experiment ran correctly:7071| Check | Result | Status |72|-------|--------|--------|73| Sample ratio mismatch (SRM) | Expected vs. actual split | Pass/Fail |74| Pre-experiment balance | Metric values before treatment | Balanced/Imbalanced |75| Novelty/primacy effects | Trend over experiment duration | Detected/Not detected |76| Interaction effects | Conflicts with concurrent experiments | None/Detected |77| Data quality | Missing data, logging issues, bot traffic | Clean/Issues found |7879### Step 5: Interpretation8081Translate statistical results into business meaning:8283| Question | Answer |84|----------|--------|85| Did the treatment work? | Yes/No/Inconclusive -- with supporting evidence |86| Is the effect practically significant? | Is the observed lift large enough to matter for the business? |87| Who benefited most? | Segment-level analysis (platform, region, user type) |88| Were there negative side effects? | Guardrail metric degradation or segment-level harm |89| What explains the result? | Causal mechanism or behavioral hypothesis |90| What don't we know? | Limitations, confounds, or unanswered questions |9192### Step 6: Recommendation9394Provide a clear recommendation:9596| Decision | Criteria |97|----------|----------|98| **Ship** | Primary metric significantly positive, no guardrail failures, practically significant lift |99| **Iterate** | Directionally positive but not significant, or significant but with guardrail concerns |100| **Do not ship** | No improvement, negative impact, or guardrail failures |101| **Extend experiment** | Insufficient power, inconclusive results, need more data |102103## Output Format104105Structure the readout for stakeholder consumption:1061071. **TL;DR** -- one-paragraph summary: what we tested, what happened, what we recommend (lead with the decision)1082. **Experiment Overview** -- hypothesis, metrics, duration, and population1093. **Results Summary** -- primary and secondary metric tables1104. **Validity Checks** -- confirmation the experiment ran correctly1115. **Segment Analysis** -- how results varied across key dimensions1126. **Interpretation** -- what the results mean for the business1137. **Recommendation** -- ship, iterate, or do not ship, with rationale1148. **Next Steps** -- follow-up experiments, monitoring plan, or rollout strategy1159. **Appendix** -- detailed statistical tables, methodology notes, and raw data references116117## Quality Checklist118119- [ ] Hypothesis is stated clearly and was defined before the experiment ran120- [ ] Primary metric was pre-registered, not chosen post-hoc121- [ ] Sample size and runtime met the pre-calculated requirements122- [ ] Validity checks (SRM, balance, novelty) have been performed123- [ ] Confidence intervals are reported alongside p-values124- [ ] Practical significance is discussed, not just statistical significance125- [ ] Guardrail metrics are checked and reported126- [ ] Segment analysis looks for heterogeneous treatment effects127- [ ] Recommendation is clear, actionable, and supported by evidence128- [ ] Limitations and caveats are honestly stated129130## Edge Cases131132- **Under-powered experiments**: Report observed effect size and confidence interval; recommend extending rather than drawing conclusions from noise133- **Multiple testing**: Apply correction methods and report both corrected and uncorrected p-values; be transparent about the number of metrics tested134- **Significant guardrail regression**: Even if the primary metric is positive, flag the guardrail failure prominently and discuss trade-offs135- **Novelty or primacy effects**: If the time-series trend shows decay or ramp-up, note this and recommend waiting for stabilization before deciding136- **Contradictory results across segments**: Report overall and segment results; do not cherry-pick favorable segments unless there is a pre-registered hypothesis for heterogeneous effects137- **Experiments stopped early**: Flag the risk of inflated effect sizes and recommend caution in interpretation
ashutoshsrivastava17