thedixitjain Holdback Experiment Design
Use this skill to plan holdbacks that preserve a comparison group after a feature rollout. Holdbacks are useful when effects may appear later, accumulate, or degrade after launch.
Source Traceability
Primary source: Practical A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from chapter 3 lines 2487-2836. Experiment-type context comes from chapter 3 lines 2013-2486.
Related Advanced Skills
long-term-impact-evaluation: use before detailed holdback design when the team should compare holdbacks against post-period analysis, continuous monitoring, CLV models, or hybrid methods.trustworthy-experiment-insights: use when deciding whether long-term or holdback evidence is credible enough to drive a product decision.
Reference Routing
| Need | Read |
|---|---|
| Holdback concepts | references/core/knowledge.md |
| Design and cost rules | references/core/rules.md |
| Scenario examples | references/core/examples.md |
| Step-by-step planning | workflows/design-holdback.md |
Workflow
- State why short-term A/B evidence is insufficient.
- Choose degradation or long-term cumulative holdback.
- Define who remains withheld, for how long, and from what experience.
- Select long-term metrics and guardrails.
- Estimate the user, business, and ethical cost of withholding.
- Define monitoring cadence, exit criteria, and communication plan.
Output Format
# Holdback Plan
## Purpose
[What long-term question this holdback answers.]
## Holdback Type
[Degradation | Long-term cumulative]
## Population And Duration
- Holdback population:
- Rollout population:
- Duration:
- Removal criteria:
## Metrics
| Metric | Role | Readout Cadence | Concern |
|--------|------|-----------------|---------|
## Cost Of Withholding
- User cost:
- Business cost:
- Ethical or trust concern:
## Decision Rules
- Continue holdback if:
- End holdback if:
- Escalate if:
Quality Bar
- Do not create a holdback without a specific long-term question.
- Do not withhold a clearly valuable feature longer than the question requires.
- Do not ignore the opportunity cost to held-back users.
- Monitor guardrails while the holdback is active.
Source: hashgraph-online/awesome-codex-plugins → plugins/LVTD-LLC/skills/skills/holdback-experiment-design/SKILL.md