Trustworthy Experiment Insights
Use this skill to decide whether an experiment result is believable enough to
shape a product or engineering decision. It focuses on false positives, false
negatives, power, replication, meta-analysis, stratified sampling, covariate
adjustment, and suspicious result review.
Source Traceability
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from Chapter 6 on false positives and negatives,
meta-analysis, metric sensitivity, stratified random sampling, covariate
adjustments, replication, longer runs, and statistical power.
Related skills:
ab-test-results-readout for standard experiment reporting.
experiment-sensitivity-optimization for improving precision before or
during experiment design.
experiment-verification-monitoring for operational validity checks.
Reference Routing
| Need |
Read |
| Insight-quality concepts |
references/core/knowledge.md |
| Credibility and follow-up rules |
references/core/rules.md |
| Result-review scenarios |
references/core/examples.md |
| Step-by-step credibility review |
workflows/review-experiment-credibility.md |
Workflow
- Confirm the experiment was operationally valid enough to interpret.
- Check power, practical significance, and whether metrics were underpowered.
- Look for false positive risk: suspicious lift, many comparisons, early stop,
weak prior, or contradiction with prior experiments.
- Look for false negative risk: noisy metrics, small sample, low sensitivity,
or over-broad metric choice.
- Compare with similar experiments or run meta-analysis when available.
- Recommend launch, replicate, extend, investigate, or reject the result.
Output Format
# Experiment Insight Credibility Review
## Result Under Review
[Experiment, metric, observed result, and proposed decision.]
## Credibility Assessment
[Trust | Trust with caveats | Replicate | Extend | Investigate | Do not trust]
## Evidence
| Check | Finding | Risk |
|-------|---------|------|
## Follow-Up
- Replication needed:
- Longer run needed:
- Meta-analysis/comparison:
- Variance reduction opportunity:
## Decision Guidance
[What decision can be made now, and what should wait.]
Quality Bar
- Do not celebrate a result before checking whether it could be a false positive.
- Do not dismiss a flat result before checking power and sensitivity.
- Do not compare against prior experiments without noting differences in
population, metric, design, and timing.
- Do not use statistical checks to hide operational failures; verify experiment
health first.
1---2name: trustworthy-experiment-insights3description: Assess whether experiment results are credible enough to influence product decisions. Use when checking false positive or false negative risk, underpowered metrics, suspiciously large lifts, replication needs, meta-analysis, stratified sampling, covariate adjustment, or whether A/B test insights should be trusted.4license: MIT5compatibility: Codex, Claude Code, and other Agent Skills-compatible clients.6metadata:7 source: https://skillmd.com/skills/hashgraph-online/trustworthy-experiment-insights8 version: "0.1.0"9 displayName: Trustworthy Experiment Insights10 category: Product Management11 tags: practical-ab-testing,next-level-ab-testing,ab-testing,experimentation,analytics12---1314# Trustworthy Experiment Insights1516Use this skill to decide whether an experiment result is believable enough to17shape a product or engineering decision. It focuses on false positives, false18negatives, power, replication, meta-analysis, stratified sampling, covariate19adjustment, and suspicious result review.2021## Source Traceability2223Primary source: *Next-Level A/B Testing* by Leemay Nassery. Guidance is24transformed and paraphrased from Chapter 6 on false positives and negatives,25meta-analysis, metric sensitivity, stratified random sampling, covariate26adjustments, replication, longer runs, and statistical power.2728Related skills:2930- `ab-test-results-readout` for standard experiment reporting.31- `experiment-sensitivity-optimization` for improving precision before or32 during experiment design.33- `experiment-verification-monitoring` for operational validity checks.3435## Reference Routing3637| Need | Read |38|------|------|39| Insight-quality concepts | `references/core/knowledge.md` |40| Credibility and follow-up rules | `references/core/rules.md` |41| Result-review scenarios | `references/core/examples.md` |42| Step-by-step credibility review | `workflows/review-experiment-credibility.md` |4344## Workflow45461. Confirm the experiment was operationally valid enough to interpret.472. Check power, practical significance, and whether metrics were underpowered.483. Look for false positive risk: suspicious lift, many comparisons, early stop,49 weak prior, or contradiction with prior experiments.504. Look for false negative risk: noisy metrics, small sample, low sensitivity,51 or over-broad metric choice.525. Compare with similar experiments or run meta-analysis when available.536. Recommend launch, replicate, extend, investigate, or reject the result.5455## Output Format5657```markdown58# Experiment Insight Credibility Review5960## Result Under Review61[Experiment, metric, observed result, and proposed decision.]6263## Credibility Assessment64[Trust | Trust with caveats | Replicate | Extend | Investigate | Do not trust]6566## Evidence67| Check | Finding | Risk |68|-------|---------|------|6970## Follow-Up71- Replication needed:72- Longer run needed:73- Meta-analysis/comparison:74- Variance reduction opportunity:7576## Decision Guidance77[What decision can be made now, and what should wait.]78```7980## Quality Bar8182- Do not celebrate a result before checking whether it could be a false positive.83- Do not dismiss a flat result before checking power and sensitivity.84- Do not compare against prior experiments without noting differences in85 population, metric, design, and timing.86- Do not use statistical checks to hide operational failures; verify experiment87 health first.