Trustworthy Experiment Insights
Use this skill to decide whether an experiment result is believable enough to
shape a product or engineering decision. It focuses on false positives, false
negatives, power, replication, meta-analysis, stratified sampling, covariate
adjustment, and suspicious result review.
Source Traceability
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from Chapter 6 on false positives and negatives,
meta-analysis, metric sensitivity, stratified random sampling, covariate
adjustments, replication, longer runs, and statistical power.
Related skills:
ab-test-results-readout for standard experiment reporting.experiment-sensitivity-optimization for improving precision before or
during experiment design.experiment-verification-monitoring for operational validity checks.
Reference Routing
| Need |
Read |
| Insight-quality concepts |
references/core/knowledge.md |
| Credibility and follow-up rules |
references/core/rules.md |
| Result-review scenarios |
references/core/examples.md |
| Step-by-step credibility review |
workflows/review-experiment-credibility.md |
Workflow
- Confirm the experiment was operationally valid enough to interpret.
- Check power, practical significance, and whether metrics were underpowered.
- Look for false positive risk: suspicious lift, many comparisons, early stop,
weak prior, or contradiction with prior experiments.
- Look for false negative risk: noisy metrics, small sample, low sensitivity,
or over-broad metric choice.
- Compare with similar experiments or run meta-analysis when available.
- Recommend launch, replicate, extend, investigate, or reject the result.
Output Format
# Experiment Insight Credibility Review
## Result Under Review
[Experiment, metric, observed result, and proposed decision.]
## Credibility Assessment
[Trust | Trust with caveats | Replicate | Extend | Investigate | Do not trust]
## Evidence
| Check | Finding | Risk |
|-------|---------|------|
## Follow-Up
- Replication needed:
- Longer run needed:
- Meta-analysis/comparison:
- Variance reduction opportunity:
## Decision Guidance
[What decision can be made now, and what should wait.]
Quality Bar
- Do not celebrate a result before checking whether it could be a false positive.
- Do not dismiss a flat result before checking power and sensitivity.
- Do not compare against prior experiments without noting differences in
population, metric, design, and timing.
- Do not use statistical checks to hide operational failures; verify experiment
health first.
1---2name: trustworthy-experiment-insights3description: Assess whether experiment results are credible enough to influence product decisions. Use when checking false positive or false negative risk, underpowered metrics, suspiciously large lifts, replication needs, meta-analysis, stratified sampling, covariate adjustment, or whether A/B test insights should be trusted.4license: MIT5---67# Trustworthy Experiment Insights89Use this skill to decide whether an experiment result is believable enough to10shape a product or engineering decision. It focuses on false positives, false11negatives, power, replication, meta-analysis, stratified sampling, covariate12adjustment, and suspicious result review.1314## Source Traceability1516Primary source: *Next-Level A/B Testing* by Leemay Nassery. Guidance is17transformed and paraphrased from Chapter 6 on false positives and negatives,18meta-analysis, metric sensitivity, stratified random sampling, covariate19adjustments, replication, longer runs, and statistical power.2021Related skills:2223- `ab-test-results-readout` for standard experiment reporting.24- `experiment-sensitivity-optimization` for improving precision before or25 during experiment design.26- `experiment-verification-monitoring` for operational validity checks.2728## Reference Routing2930| Need | Read |31|------|------|32| Insight-quality concepts | `references/core/knowledge.md` |33| Credibility and follow-up rules | `references/core/rules.md` |34| Result-review scenarios | `references/core/examples.md` |35| Step-by-step credibility review | `workflows/review-experiment-credibility.md` |3637## Workflow38391. Confirm the experiment was operationally valid enough to interpret.402. Check power, practical significance, and whether metrics were underpowered.413. Look for false positive risk: suspicious lift, many comparisons, early stop,42 weak prior, or contradiction with prior experiments.434. Look for false negative risk: noisy metrics, small sample, low sensitivity,44 or over-broad metric choice.455. Compare with similar experiments or run meta-analysis when available.466. Recommend launch, replicate, extend, investigate, or reject the result.4748## Output Format4950```markdown51# Experiment Insight Credibility Review5253## Result Under Review54[Experiment, metric, observed result, and proposed decision.]5556## Credibility Assessment57[Trust | Trust with caveats | Replicate | Extend | Investigate | Do not trust]5859## Evidence60| Check | Finding | Risk |61|-------|---------|------|6263## Follow-Up64- Replication needed:65- Longer run needed:66- Meta-analysis/comparison:67- Variance reduction opportunity:6869## Decision Guidance70[What decision can be made now, and what should wait.]71```7273## Quality Bar7475- Do not celebrate a result before checking whether it could be a false positive.76- Do not dismiss a flat result before checking power and sensitivity.77- Do not compare against prior experiments without noting differences in78 population, metric, design, and timing.79- Do not use statistical checks to hide operational failures; verify experiment80 health first.
LVTD-LLC