Trustworthy Experiment Insights

Trustworthy Experiment Insights

Use this skill to decide whether an experiment result is believable enough to shape a product or engineering decision. It focuses on false positives, false negatives, power, replication, meta-analysis, stratified sampling, covariate adjustment, and suspicious result review.

Source Traceability

Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from Chapter 6 on false positives and negatives, meta-analysis, metric sensitivity, stratified random sampling, covariate adjustments, replication, longer runs, and statistical power.

Related skills:

• ab-test-results-readout for standard experiment reporting.
• experiment-sensitivity-optimization for improving precision before or during experiment design.
• experiment-verification-monitoring for operational validity checks.

Reference Routing

Workflow

1. Confirm the experiment was operationally valid enough to interpret.
2. Check power, practical significance, and whether metrics were underpowered.
3. Look for false positive risk: suspicious lift, many comparisons, early stop, weak prior, or contradiction with prior experiments.
4. Look for false negative risk: noisy metrics, small sample, low sensitivity, or over-broad metric choice.
5. Compare with similar experiments or run meta-analysis when available.
6. Recommend launch, replicate, extend, investigate, or reject the result.

Output Format

# Experiment Insight Credibility Review

## Result Under Review
[Experiment, metric, observed result, and proposed decision.]

## Credibility Assessment
[Trust | Trust with caveats | Replicate | Extend | Investigate | Do not trust]

## Evidence
| Check | Finding | Risk |
|-------|---------|------|

## Follow-Up
- Replication needed:
- Longer run needed:
- Meta-analysis/comparison:
- Variance reduction opportunity:

## Decision Guidance
[What decision can be made now, and what should wait.]

Quality Bar

• Do not celebrate a result before checking whether it could be a false positive.
• Do not dismiss a flat result before checking power and sensitivity.
• Do not compare against prior experiments without noting differences in population, metric, design, and timing.
• Do not use statistical checks to hide operational failures; verify experiment health first.