Experiment Sensitivity Optimization
Use this skill to redesign an experiment so it can detect meaningful effects
with fewer users, less time, or clearer metrics. It focuses on minimum
detectable effect, metric sensitivity, capping, variant reduction, CUPED, and
variance reduction.
Source Traceability
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from Chapter 3 on experiment design, sensitive
metrics, minimum detectable effect, capping, reducing variants, and CUPED; and
Chapter 6 on stratified random sampling and covariate adjustments.
Related skills:
ab-test-design-brief for baseline experiment specs.
trustworthy-experiment-insights for judging whether a result is believable.
experimentation-throughput-strategy for capacity and test scheduling.
Reference Routing
| Need |
Read |
| Sensitivity concepts |
references/core/knowledge.md |
| Metric, variance, and sample-size rules |
references/core/rules.md |
| Optimization scenarios |
references/core/examples.md |
| Step-by-step sensitivity review |
workflows/optimize-experiment-sensitivity.md |
Workflow
- State the decision and the smallest practically meaningful effect.
- Check whether the current primary metric is close enough to the feature's
mechanism.
- Reduce unnecessary variants and separate learning tests from launch tests.
- Consider metric capping, CUPED, stratification, or other variance reduction.
- Record data prerequisites, risks, and interpretation limits.
- Update the experiment brief with the revised measurement plan.
Output Format
# Experiment Sensitivity Plan
## Decision
[What the experiment must decide.]
## Current Constraint
[Traffic | Duration | Noisy metric | Too many variants | Weak proxy | Other]
## Recommended Changes
| Change | Why It Helps | Requirement | Risk |
|--------|--------------|-------------|------|
## Metric Plan
- Primary metric:
- More sensitive alternative:
- Guardrails:
- Minimum detectable effect:
## Variance Reduction
- Technique:
- Data needed:
- Validation:
## Interpretation Notes
- What this design can conclude:
- What it cannot conclude:
Quality Bar
- Do not optimize sensitivity by switching to a metric that no longer answers
the product decision.
- Do not add CUPED, stratification, or capping unless the data requirements and
interpretation risks are named.
- Do not keep extra variants when they are not needed for the decision.
- Do not treat a smaller detectable effect as useful unless it is practically
meaningful.
1---2name: experiment-sensitivity-optimization3description: Improve experiment sensitivity and reduce traffic or duration requirements. Use when choosing sensitive metrics, working with minimum detectable effect, reducing variants, applying capping metrics, CUPED, variance reduction, or deciding how to get trustworthy A/B test signal with fewer users.4license: MIT5compatibility: Codex, Claude Code, and other Agent Skills-compatible clients.6metadata:7 source: https://skillmd.com/skills/hashgraph-online/experiment-sensitivity-optimization8 version: "0.1.0"9 displayName: Experiment Sensitivity Optimization10 category: Product Management11 tags: practical-ab-testing,next-level-ab-testing,ab-testing,experimentation,metrics12---1314# Experiment Sensitivity Optimization1516Use this skill to redesign an experiment so it can detect meaningful effects17with fewer users, less time, or clearer metrics. It focuses on minimum18detectable effect, metric sensitivity, capping, variant reduction, CUPED, and19variance reduction.2021## Source Traceability2223Primary source: *Next-Level A/B Testing* by Leemay Nassery. Guidance is24transformed and paraphrased from Chapter 3 on experiment design, sensitive25metrics, minimum detectable effect, capping, reducing variants, and CUPED; and26Chapter 6 on stratified random sampling and covariate adjustments.2728Related skills:2930- `ab-test-design-brief` for baseline experiment specs.31- `trustworthy-experiment-insights` for judging whether a result is believable.32- `experimentation-throughput-strategy` for capacity and test scheduling.3334## Reference Routing3536| Need | Read |37|------|------|38| Sensitivity concepts | `references/core/knowledge.md` |39| Metric, variance, and sample-size rules | `references/core/rules.md` |40| Optimization scenarios | `references/core/examples.md` |41| Step-by-step sensitivity review | `workflows/optimize-experiment-sensitivity.md` |4243## Workflow44451. State the decision and the smallest practically meaningful effect.462. Check whether the current primary metric is close enough to the feature's47 mechanism.483. Reduce unnecessary variants and separate learning tests from launch tests.494. Consider metric capping, CUPED, stratification, or other variance reduction.505. Record data prerequisites, risks, and interpretation limits.516. Update the experiment brief with the revised measurement plan.5253## Output Format5455```markdown56# Experiment Sensitivity Plan5758## Decision59[What the experiment must decide.]6061## Current Constraint62[Traffic | Duration | Noisy metric | Too many variants | Weak proxy | Other]6364## Recommended Changes65| Change | Why It Helps | Requirement | Risk |66|--------|--------------|-------------|------|6768## Metric Plan69- Primary metric:70- More sensitive alternative:71- Guardrails:72- Minimum detectable effect:7374## Variance Reduction75- Technique:76- Data needed:77- Validation:7879## Interpretation Notes80- What this design can conclude:81- What it cannot conclude:82```8384## Quality Bar8586- Do not optimize sensitivity by switching to a metric that no longer answers87 the product decision.88- Do not add CUPED, stratification, or capping unless the data requirements and89 interpretation risks are named.90- Do not keep extra variants when they are not needed for the decision.91- Do not treat a smaller detectable effect as useful unless it is practically92 meaningful.