Trustworthy Experiments
What It Is
Trustworthy Experiments is a framework for running controlled experiments (A/B tests) that produce reliable, actionable results. The core insight: most experiments fail, and many "successful" results are actually false positives.
The key shift: Move from "Did the experiment show a positive result?" to "Can I trust this result enough to act on it?"
Ronny Kohavi, who built experimentation platforms at Microsoft, Amazon, and Airbnb, found that:
- 66-92% of experiments fail to improve the target metric
- 8% of experiments have invalid results due to sample ratio mismatch alone
- When the base success rate is 8%, a P-value of 0.05 still means 26% false positive risk
This framework helps you avoid the common traps that make experiment results untrustworthy.
Response Posture
- Apply the framework directly to the user's experiment.
- Never mention the repository, skills, SKILL.md, patterns, or references.
- Do not run tools or read files; answer from the framework.
- Avoid process/meta commentary; respond as an experimentation lead.
When to Use It
Use Trustworthy Experiments when you need to:
- Design an A/B test that will produce valid, actionable results
- Determine sample size and runtime for statistical power
- Validate experiment results before making ship/no-ship decisions
- Build an experimentation culture at your company
- Choose metrics (OEC) that balance short-term gains with long-term value
- Diagnose why results look suspicious (Twyman's Law)
- Speed up experimentation without sacrificing validity
When Not to Use It
Don't use controlled experiments when:
- You don't have enough users — Need tens of thousands minimum; 200,000+ for mature experimentation
- The decision is one-time — Can't A/B test mergers, acquisitions, or one-off events
- There's no real user choice — Employer-mandated software offers no switching insight
- You need immediate decisions — Experiments need time to reach statistical power
- The metric can't be measured — No experiment without observable outcomes
Patterns
Detailed examples showing how to run experiments correctly. Each pattern shows a common mistake and the correct approach.
Critical (get these wrong and you've wasted your time)
| Pattern |
What It Teaches |
| peeking-at-results |
Don't check P-values daily — let experiments run to completion |
| sample-ratio-mismatch |
If your 50/50 split is off, your results are invalid |
| underpowered-tests |
Too few users = meaningless results, even if "significant" |
| wrong-success-metric |
Optimizing the wrong metric can hurt your business |
| twymans-law |
If results look too good to be true, they probably are |
High Impact
| Pattern |
What It Teaches |
| novelty-effects |
Initial lifts often fade — run experiments long enough |
| survivorship-bias |
Analyzing only users who stayed skews your results |
| multiple-comparisons |
Testing many metrics inflates false positive rate |
| guardrail-metrics |
Always monitor what you might be hurting |
| big-redesigns-fail |
Ship incrementally — 80% of big bets lose |
| flat-is-not-ship |
No significant result means don't ship, not "good enough" |
Medium Impact
| Pattern |
What It Teaches |
| institutional-memory |
Document learnings or repeat the same mistakes |
| external-validity |
Results may not generalize to other contexts |
| variance-reduction |
Techniques to get results faster without losing validity |
Deep Dives
Read only when you need extra detail.
references/trustworthy-experiments-playbook.md: Expanded framework detail, checklists, and examples.references/experiment-plan-template.md: Fill-in-the-blanks plan to design and run an A/B test.
Scripts
Optional utilities (no external deps):
scripts/sample_size.py: Estimate required sample size for a two-variant conversion test.scripts/srm_check.py: Check sample ratio mismatch (SRM) for a 2-bucket split.
Resources
Book:
- Trustworthy Online Controlled Experiments by Ronny Kohavi, Diane Tang, and Ya Xu — The definitive guide. All proceeds go to charity.
Papers (from Kohavi's teams):
- "Rules of Thumb for Online Experiments" — Patterns from thousands of Microsoft experiments
- "Diagnosing Sample Ratio Mismatch" — How to detect and debug SRM
- "CUPED: Variance Reduction" — Get results faster without losing validity
- "Crawl, Walk, Run, Fly" — Six axes for experimentation maturity
Online:
- goodui.org — Database of 140+ experiment patterns with success rates
- Ronny Kohavi's LinkedIn — Regular posts on experimentation insights
- Ronny Kohavi's Maven course — Live cohort-based course on experimentation
Related Books:
- Calling Bullshit by Carl Bergstrom and Jevin West — Critical thinking about data
- Hard Facts, Dangerous Half-Truths and Total Nonsense by Jeffrey Pfeffer and Robert Sutton — Evidence-based management
1---2name: trustworthy-experiments3description: Use when asked to "run an A/B test", "design an experiment", "check statistical significance", "trust our results", "avoid false positives", or "experiment guardrails". Helps design, run, and interpret controlled experiments correctly. Based on Ronny Kohavi's framework from "Trustworthy Online Controlled Experiments".4---56# Trustworthy Experiments78## What It Is910Trustworthy Experiments is a framework for running controlled experiments (A/B tests) that produce reliable, actionable results. The core insight: **most experiments fail, and many "successful" results are actually false positives.**1112The key shift: Move from "Did the experiment show a positive result?" to "Can I trust this result enough to act on it?"1314Ronny Kohavi, who built experimentation platforms at Microsoft, Amazon, and Airbnb, found that:15- **66-92% of experiments fail** to improve the target metric16- **8% of experiments have invalid results** due to sample ratio mismatch alone17- When the base success rate is 8%, a P-value of 0.05 still means **26% false positive risk**1819This framework helps you avoid the common traps that make experiment results untrustworthy.2021## Response Posture2223- Apply the framework directly to the user's experiment.24- Never mention the repository, skills, SKILL.md, patterns, or references.25- Do not run tools or read files; answer from the framework.26- Avoid process/meta commentary; respond as an experimentation lead.2728## When to Use It2930Use Trustworthy Experiments when you need to:3132- **Design an A/B test** that will produce valid, actionable results33- **Determine sample size and runtime** for statistical power34- **Validate experiment results** before making ship/no-ship decisions35- **Build an experimentation culture** at your company36- **Choose metrics (OEC)** that balance short-term gains with long-term value37- **Diagnose why results look suspicious** (Twyman's Law)38- **Speed up experimentation** without sacrificing validity3940## When Not to Use It4142Don't use controlled experiments when:4344- **You don't have enough users** — Need tens of thousands minimum; 200,000+ for mature experimentation45- **The decision is one-time** — Can't A/B test mergers, acquisitions, or one-off events46- **There's no real user choice** — Employer-mandated software offers no switching insight47- **You need immediate decisions** — Experiments need time to reach statistical power48- **The metric can't be measured** — No experiment without observable outcomes4950## Patterns5152Detailed examples showing how to run experiments correctly. Each pattern shows a common mistake and the correct approach.5354### Critical (get these wrong and you've wasted your time)5556| Pattern | What It Teaches |57|---------|-----------------|58| [peeking-at-results](patterns/peeking-at-results.md) | Don't check P-values daily — let experiments run to completion |59| [sample-ratio-mismatch](patterns/sample-ratio-mismatch.md) | If your 50/50 split is off, your results are invalid |60| [underpowered-tests](patterns/underpowered-tests.md) | Too few users = meaningless results, even if "significant" |61| [wrong-success-metric](patterns/wrong-success-metric.md) | Optimizing the wrong metric can hurt your business |62| [twymans-law](patterns/twymans-law.md) | If results look too good to be true, they probably are |6364### High Impact6566| Pattern | What It Teaches |67|---------|-----------------|68| [novelty-effects](patterns/novelty-effects.md) | Initial lifts often fade — run experiments long enough |69| [survivorship-bias](patterns/survivorship-bias.md) | Analyzing only users who stayed skews your results |70| [multiple-comparisons](patterns/multiple-comparisons.md) | Testing many metrics inflates false positive rate |71| [guardrail-metrics](patterns/guardrail-metrics.md) | Always monitor what you might be hurting |72| [big-redesigns-fail](patterns/big-redesigns-fail.md) | Ship incrementally — 80% of big bets lose |73| [flat-is-not-ship](patterns/flat-is-not-ship.md) | No significant result means don't ship, not "good enough" |7475### Medium Impact7677| Pattern | What It Teaches |78|---------|-----------------|79| [institutional-memory](patterns/institutional-memory.md) | Document learnings or repeat the same mistakes |80| [external-validity](patterns/external-validity.md) | Results may not generalize to other contexts |81| [variance-reduction](patterns/variance-reduction.md) | Techniques to get results faster without losing validity |828384## Deep Dives8586Read only when you need extra detail.8788- `references/trustworthy-experiments-playbook.md`: Expanded framework detail, checklists, and examples.89- `references/experiment-plan-template.md`: Fill-in-the-blanks plan to design and run an A/B test.9091## Scripts9293Optional utilities (no external deps):9495- `scripts/sample_size.py`: Estimate required sample size for a two-variant conversion test.96- `scripts/srm_check.py`: Check sample ratio mismatch (SRM) for a 2-bucket split.9798## Resources99100**Book:**101- *Trustworthy Online Controlled Experiments* by Ronny Kohavi, Diane Tang, and Ya Xu — The definitive guide. All proceeds go to charity.102103**Papers (from Kohavi's teams):**104- "Rules of Thumb for Online Experiments" — Patterns from thousands of Microsoft experiments105- "Diagnosing Sample Ratio Mismatch" — How to detect and debug SRM106- "CUPED: Variance Reduction" — Get results faster without losing validity107- "Crawl, Walk, Run, Fly" — Six axes for experimentation maturity108109**Online:**110- goodui.org — Database of 140+ experiment patterns with success rates111- Ronny Kohavi's LinkedIn — Regular posts on experimentation insights112- Ronny Kohavi's Maven course — Live cohort-based course on experimentation113114**Related Books:**115- *Calling Bullshit* by Carl Bergstrom and Jevin West — Critical thinking about data116- *Hard Facts, Dangerous Half-Truths and Total Nonsense* by Jeffrey Pfeffer and Robert Sutton — Evidence-based management
wdavidturner