Stijnman Experiment Results Analysis
Purpose
Create an honest experiment readout that separates measured outcomes from interpretation and avoids treating inconclusive data as a decision.
Workflow
- Record the hypothesis, variants, primary metric, guardrails, sample sizes, time window, and pre-defined stopping rule when available.
- Check data quality, assignment integrity, missing values, and material deviations from the experiment plan.
- Report primary and guardrail metrics with effect estimates, uncertainty, and the method used.
- Review relevant segments cautiously; label exploratory segment findings and avoid overinterpreting small samples.
- State whether the evidence supports shipping, iterating, stopping, or gathering more data.
- Capture learnings, limitations, and next steps.
Evidence boundaries
Do not invent baselines, statistical significance, causal conclusions, or business impact. A decision recommendation must state the evidence strength and any remaining risk.
Error handling
| Situation | Response |
|---|---|
| Required metrics are missing | Describe the gap and recommend a measurement plan. |
| Results are inconclusive | State that clearly and propose the smallest useful follow-up. |
| Guardrail degrades | Flag the trade-off even if the primary metric improves. |
| Segments are underpowered | Treat them as exploratory rather than decisive. |