jeremylongworth-source Experiment Readout
Core Workflow
- Identify hypothesis, variants, audience, dates, primary metric, guardrails, sample, and decision rule.
- Summarize results against the pre-stated decision rule.
- Separate observed movement, statistical/practical significance, data quality, and interpretation.
- Check guardrails and unintended effects before recommending rollout.
- State decision: ship, iterate, stop, extend, or rerun.
- Capture learning, follow-up questions, and next experiment.
Safety Rules
- Do not invent sample sizes, statistical significance, confidence intervals, or metric values.
- Do not recommend rollout when guardrails failed or data quality is poor.
- Escalate pricing, medical, legal, employment, financial, or high-risk customer-impacting experiments.
Deliverable Shape
For experiment readouts, provide:
- Hypothesis and setup
- Metrics and decision rule
- Results summary
- Guardrail review
- Data quality caveats
- Decision
- Learning
- Follow-up actions
References
- Read
references/experiment-readout-checklist.mdwhen writing experiment or pilot readouts.