itsual Experiment Design
Overview
Experiments turn opinions into evidence. Good experiments have a clear hypothesis, a sensible design, and pre-committed success criteria.
When to Use
- Testing feature or UX changes
- Evaluating growth, pricing, or messaging ideas
- Reducing uncertainty before full rollout
- Measuring incremental impact of product changes
Core Elements
- Hypothesis and rationale
- Primary metric and guardrail metrics
- Target population and sample size / duration reasoning
- Variants and what differs between them
- Randomization and assignment method
- Success criteria and decision rules
- Risks and mitigations
Principles
- Decide how you will interpret results before looking at them
- Don’t peek and stop early without a plan
- Watch for novelty effects, seasonality, and interference
- Prefer fewer, well-powered experiments over many inconclusive ones
- Document outcomes and share learning even when the test “loses”
Verification
- Hypothesis and metrics are explicit
- Experiment can actually produce a decision
- Results (including null results) are recorded