Trustworthy Experiment Insights
Use this skill to decide whether an experiment result is believable enough to
shape a product or engineering decision. It focuses on false positives, false
negatives, power, replication, meta-analysis, stratified sampling, covariate
adjustment, and suspicious result review.
Source Traceability
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from Chapter 6 on false positives and negatives,
meta-analysis, metric sensitivity, stratified random sampling, covariate
adjustments, replication, longer runs, and statistical power.
Related skills:
ab-test-results-readout for standard experiment reporting.experiment-sensitivity-optimization for improving precision before or
during experiment design.experiment-verification-monitoring for operational validity checks.
Reference Routing
| Need |
Read |
| Insight-quality concepts |
references/core/knowledge.md |
| Credibility and follow-up rules |
references/core/rules.md |
| Result-review scenarios |
references/core/examples.md |
| Step-by-step credibility review |
workflows/review-experiment-credibility.md |
Workflow
- Confirm the experiment was operationally valid enough to interpret.
- Check power, practical significance, and whether metrics were underpowered.
- Look for false positive risk: suspicious lift, many comparisons, early stop,
weak prior, or contradiction with prior experiments.
- Look for false negative risk: noisy metrics, small sample, low sensitivity,
or over-broad metric choice.
- Compare with similar experiments or run meta-analysis when available.
- Recommend launch, replicate, extend, investigate, or reject the result.
Output Format
# Experiment Insight Credibility Review
## Result Under Review
[Experiment, metric, observed result, and proposed decision.]
## Credibility Assessment
[Trust | Trust with caveats | Replicate | Extend | Investigate | Do not trust]
## Evidence
| Check | Finding | Risk |
|-------|---------|------|
## Follow-Up
- Replication needed:
- Longer run needed:
- Meta-analysis/comparison:
- Variance reduction opportunity:
## Decision Guidance
[What decision can be made now, and what should wait.]
Quality Bar
- Do not celebrate a result before checking whether it could be a false positive.
- Do not dismiss a flat result before checking power and sensitivity.
- Do not compare against prior experiments without noting differences in
population, metric, design, and timing.
- Do not use statistical checks to hide operational failures; verify experiment
health first.
Source: hashgraph-online/awesome-codex-plugins → plugins/LVTD-LLC/skills/skills/trustworthy-experiment-insights/SKILL.md
1---2name: trustworthy-experiment-insights3description: Assess whether experiment results are credible enough to influence product decisions. Use when checking false positive or false negative risk, underpowered metrics, suspiciously large lifts, replication needs, meta-analysis, stratified sampling, covariate adjustment, or whether A/B test insights should be trusted.4---5
6
7# Trustworthy Experiment Insights
8
9Use this skill to decide whether an experiment result is believable enough to
10shape a product or engineering decision. It focuses on false positives, false
11negatives, power, replication, meta-analysis, stratified sampling, covariate
12adjustment, and suspicious result review.
13
14## Source Traceability
15
16Primary source: *Next-Level A/B Testing* by Leemay Nassery. Guidance is
17transformed and paraphrased from Chapter 6 on false positives and negatives,
18meta-analysis, metric sensitivity, stratified random sampling, covariate
19adjustments, replication, longer runs, and statistical power.
20
21Related skills:
22
23- `ab-test-results-readout` for standard experiment reporting.
24- `experiment-sensitivity-optimization` for improving precision before or
25 during experiment design.
26- `experiment-verification-monitoring` for operational validity checks.
27
28## Reference Routing
29
30| Need | Read |
31|------|------|
32| Insight-quality concepts | `references/core/knowledge.md` |
33| Credibility and follow-up rules | `references/core/rules.md` |
34| Result-review scenarios | `references/core/examples.md` |
35| Step-by-step credibility review | `workflows/review-experiment-credibility.md` |
36
37## Workflow
38
391. Confirm the experiment was operationally valid enough to interpret.
402. Check power, practical significance, and whether metrics were underpowered.
413. Look for false positive risk: suspicious lift, many comparisons, early stop,
42 weak prior, or contradiction with prior experiments.
434. Look for false negative risk: noisy metrics, small sample, low sensitivity,
44 or over-broad metric choice.
455. Compare with similar experiments or run meta-analysis when available.
466. Recommend launch, replicate, extend, investigate, or reject the result.
47
48## Output Format
49
50```markdown
51# Experiment Insight Credibility Review
52
53## Result Under Review
54[Experiment, metric, observed result, and proposed decision.]
55
56## Credibility Assessment
57[Trust | Trust with caveats | Replicate | Extend | Investigate | Do not trust]
58
59## Evidence
60| Check | Finding | Risk |
61|-------|---------|------|
62
63## Follow-Up
64- Replication needed:
65- Longer run needed:
66- Meta-analysis/comparison:
67- Variance reduction opportunity:
68
69## Decision Guidance
70[What decision can be made now, and what should wait.]
71```
72
73## Quality Bar
74
75- Do not celebrate a result before checking whether it could be a false positive.
76- Do not dismiss a flat result before checking power and sensitivity.
77- Do not compare against prior experiments without noting differences in
78 population, metric, design, and timing.
79- Do not use statistical checks to hide operational failures; verify experiment
80 health first.
81
82---
83
84**Source:** [`hashgraph-online/awesome-codex-plugins`](https://github.com/hashgraph-online/awesome-codex-plugins) → `plugins/LVTD-LLC/skills/skills/trustworthy-experiment-insights/SKILL.md`
thedixitjain