Holdback Experiment Design

Holdback Experiment Design

Use this skill to plan holdbacks that preserve a comparison group after a feature rollout. Holdbacks are useful when effects may appear later, accumulate, or degrade after launch.

Source Traceability

Primary source: Practical A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from chapter 3 lines 2487-2836. Experiment-type context comes from chapter 3 lines 2013-2486.

Related Advanced Skills

• long-term-impact-evaluation: use before detailed holdback design when the team should compare holdbacks against post-period analysis, continuous monitoring, CLV models, or hybrid methods.
• trustworthy-experiment-insights: use when deciding whether long-term or holdback evidence is credible enough to drive a product decision.

Reference Routing

Workflow

1. State why short-term A/B evidence is insufficient.
2. Choose degradation or long-term cumulative holdback.
3. Define who remains withheld, for how long, and from what experience.
4. Select long-term metrics and guardrails.
5. Estimate the user, business, and ethical cost of withholding.
6. Define monitoring cadence, exit criteria, and communication plan.

Output Format

# Holdback Plan

## Purpose
[What long-term question this holdback answers.]

## Holdback Type
[Degradation | Long-term cumulative]

## Population And Duration
- Holdback population:
- Rollout population:
- Duration:
- Removal criteria:

## Metrics
| Metric | Role | Readout Cadence | Concern |
|--------|------|-----------------|---------|

## Cost Of Withholding
- User cost:
- Business cost:
- Ethical or trust concern:

## Decision Rules
- Continue holdback if:
- End holdback if:
- Escalate if:

Quality Bar

• Do not create a holdback without a specific long-term question.
• Do not withhold a clearly valuable feature longer than the question requires.
• Do not ignore the opportunity cost to held-back users.
• Monitor guardrails while the holdback is active.