Adaptive Experimentation Strategy
Use this skill to decide whether and how to use adaptive testing strategies
instead of fixed-horizon A/B tests. It covers sequential testing, multi-armed
bandits, Thompson sampling, contextual bandits, exploration/exploitation, and
the data and engineering readiness required to operate them.
Source Traceability
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from Chapter 7 on adaptive testing, sequential
testing, multi-armed bandits, Thompson sampling, contextual bandits, and
engineering requirements.
Related skills:
experiment-type-selection for choosing simpler experiment types first.
ml-experiment-evaluation for model and ranking evaluation paths.
experiment-verification-monitoring for operational health and alerting.
Reference Routing
| Need |
Read |
| Adaptive testing concepts |
references/core/knowledge.md |
| Readiness and strategy rules |
references/core/rules.md |
| Scenario examples |
references/core/examples.md |
| Step-by-step adaptive readiness plan |
workflows/evaluate-adaptive-strategy.md |
Workflow
- State the decision and why fixed-horizon A/B testing may not be enough.
- Decide whether the need is early stopping, reward maximization, or
personalization.
- Check data freshness, reward definition, dashboards, on-call ownership, and
rollback controls.
- Choose sequential testing, bandits, Thompson sampling, contextual bandits, or
a simpler alternative.
- Document exploration/exploitation tradeoffs and user/business risk.
- Define rollout, monitoring, and adoption requirements.
Output Format
# Adaptive Experimentation Recommendation
## Use Case
[What decision or allocation problem motivates adaptive testing.]
## Recommended Strategy
[Do not use adaptive testing | Sequential | Bandit | Thompson sampling | Contextual bandit]
## Readiness
| Requirement | Status | Gap |
|-------------|--------|-----|
## Tradeoffs
- Reward:
- Exploration cost:
- Data freshness:
- Operational risk:
## Rollout Plan
1. [Step]
2. [Step]
3. [Step]
Quality Bar
- Do not recommend adaptive testing just because it is advanced.
- Do not use bandits when the real need is a clean causal estimate.
- Do not use contextual bandits without reliable context features and reward
measurement.
- Do not ignore production requirements: stale data, bad allocation, and alert
ownership can break adaptive systems.
1---2name: adaptive-experimentation-strategy3description: Plan adaptive experimentation strategies beyond fixed-horizon A/B tests. Use when evaluating sequential testing, early stopping, multi-armed bandits, Thompson sampling, contextual bandits, dynamic traffic allocation, exploration/exploitation tradeoffs, or readiness for adaptive testing infrastructure.4license: MIT5compatibility: Codex, Claude Code, and other Agent Skills-compatible clients.6metadata:7 source: https://skillmd.com/skills/hashgraph-online/adaptive-experimentation-strategy8 version: "0.1.0"9 displayName: Adaptive Experimentation Strategy10 category: Product Management11 tags: practical-ab-testing,next-level-ab-testing,ab-testing,experimentation,adaptive-testing12---1314# Adaptive Experimentation Strategy1516Use this skill to decide whether and how to use adaptive testing strategies17instead of fixed-horizon A/B tests. It covers sequential testing, multi-armed18bandits, Thompson sampling, contextual bandits, exploration/exploitation, and19the data and engineering readiness required to operate them.2021## Source Traceability2223Primary source: *Next-Level A/B Testing* by Leemay Nassery. Guidance is24transformed and paraphrased from Chapter 7 on adaptive testing, sequential25testing, multi-armed bandits, Thompson sampling, contextual bandits, and26engineering requirements.2728Related skills:2930- `experiment-type-selection` for choosing simpler experiment types first.31- `ml-experiment-evaluation` for model and ranking evaluation paths.32- `experiment-verification-monitoring` for operational health and alerting.3334## Reference Routing3536| Need | Read |37|------|------|38| Adaptive testing concepts | `references/core/knowledge.md` |39| Readiness and strategy rules | `references/core/rules.md` |40| Scenario examples | `references/core/examples.md` |41| Step-by-step adaptive readiness plan | `workflows/evaluate-adaptive-strategy.md` |4243## Workflow44451. State the decision and why fixed-horizon A/B testing may not be enough.462. Decide whether the need is early stopping, reward maximization, or47 personalization.483. Check data freshness, reward definition, dashboards, on-call ownership, and49 rollback controls.504. Choose sequential testing, bandits, Thompson sampling, contextual bandits, or51 a simpler alternative.525. Document exploration/exploitation tradeoffs and user/business risk.536. Define rollout, monitoring, and adoption requirements.5455## Output Format5657```markdown58# Adaptive Experimentation Recommendation5960## Use Case61[What decision or allocation problem motivates adaptive testing.]6263## Recommended Strategy64[Do not use adaptive testing | Sequential | Bandit | Thompson sampling | Contextual bandit]6566## Readiness67| Requirement | Status | Gap |68|-------------|--------|-----|6970## Tradeoffs71- Reward:72- Exploration cost:73- Data freshness:74- Operational risk:7576## Rollout Plan771. [Step]782. [Step]793. [Step]80```8182## Quality Bar8384- Do not recommend adaptive testing just because it is advanced.85- Do not use bandits when the real need is a clean causal estimate.86- Do not use contextual bandits without reliable context features and reward87 measurement.88- Do not ignore production requirements: stale data, bad allocation, and alert89 ownership can break adaptive systems.