Adaptive Experimentation Strategy
Use this skill to decide whether and how to use adaptive testing strategies
instead of fixed-horizon A/B tests. It covers sequential testing, multi-armed
bandits, Thompson sampling, contextual bandits, exploration/exploitation, and
the data and engineering readiness required to operate them.
Source Traceability
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from Chapter 7 on adaptive testing, sequential
testing, multi-armed bandits, Thompson sampling, contextual bandits, and
engineering requirements.
Related skills:
experiment-type-selection for choosing simpler experiment types first.ml-experiment-evaluation for model and ranking evaluation paths.experiment-verification-monitoring for operational health and alerting.
Reference Routing
| Need |
Read |
| Adaptive testing concepts |
references/core/knowledge.md |
| Readiness and strategy rules |
references/core/rules.md |
| Scenario examples |
references/core/examples.md |
| Step-by-step adaptive readiness plan |
workflows/evaluate-adaptive-strategy.md |
Workflow
- State the decision and why fixed-horizon A/B testing may not be enough.
- Decide whether the need is early stopping, reward maximization, or
personalization.
- Check data freshness, reward definition, dashboards, on-call ownership, and
rollback controls.
- Choose sequential testing, bandits, Thompson sampling, contextual bandits, or
a simpler alternative.
- Document exploration/exploitation tradeoffs and user/business risk.
- Define rollout, monitoring, and adoption requirements.
Output Format
# Adaptive Experimentation Recommendation
## Use Case
[What decision or allocation problem motivates adaptive testing.]
## Recommended Strategy
[Do not use adaptive testing | Sequential | Bandit | Thompson sampling | Contextual bandit]
## Readiness
| Requirement | Status | Gap |
|-------------|--------|-----|
## Tradeoffs
- Reward:
- Exploration cost:
- Data freshness:
- Operational risk:
## Rollout Plan
1. [Step]
2. [Step]
3. [Step]
Quality Bar
- Do not recommend adaptive testing just because it is advanced.
- Do not use bandits when the real need is a clean causal estimate.
- Do not use contextual bandits without reliable context features and reward
measurement.
- Do not ignore production requirements: stale data, bad allocation, and alert
ownership can break adaptive systems.
Source: hashgraph-online/awesome-codex-plugins → plugins/LVTD-LLC/skills/skills/adaptive-experimentation-strategy/SKILL.md
1---2name: adaptive-experimentation-strategy3description: Plan adaptive experimentation strategies beyond fixed-horizon A/B tests. Use when evaluating sequential testing, early stopping, multi-armed bandits, Thompson sampling, contextual bandits, dynamic traffic allocation, exploration/exploitation tradeoffs, or readiness for adaptive testing infrastructure.4---5
6
7# Adaptive Experimentation Strategy
8
9Use this skill to decide whether and how to use adaptive testing strategies
10instead of fixed-horizon A/B tests. It covers sequential testing, multi-armed
11bandits, Thompson sampling, contextual bandits, exploration/exploitation, and
12the data and engineering readiness required to operate them.
13
14## Source Traceability
15
16Primary source: *Next-Level A/B Testing* by Leemay Nassery. Guidance is
17transformed and paraphrased from Chapter 7 on adaptive testing, sequential
18testing, multi-armed bandits, Thompson sampling, contextual bandits, and
19engineering requirements.
20
21Related skills:
22
23- `experiment-type-selection` for choosing simpler experiment types first.
24- `ml-experiment-evaluation` for model and ranking evaluation paths.
25- `experiment-verification-monitoring` for operational health and alerting.
26
27## Reference Routing
28
29| Need | Read |
30|------|------|
31| Adaptive testing concepts | `references/core/knowledge.md` |
32| Readiness and strategy rules | `references/core/rules.md` |
33| Scenario examples | `references/core/examples.md` |
34| Step-by-step adaptive readiness plan | `workflows/evaluate-adaptive-strategy.md` |
35
36## Workflow
37
381. State the decision and why fixed-horizon A/B testing may not be enough.
392. Decide whether the need is early stopping, reward maximization, or
40 personalization.
413. Check data freshness, reward definition, dashboards, on-call ownership, and
42 rollback controls.
434. Choose sequential testing, bandits, Thompson sampling, contextual bandits, or
44 a simpler alternative.
455. Document exploration/exploitation tradeoffs and user/business risk.
466. Define rollout, monitoring, and adoption requirements.
47
48## Output Format
49
50```markdown
51# Adaptive Experimentation Recommendation
52
53## Use Case
54[What decision or allocation problem motivates adaptive testing.]
55
56## Recommended Strategy
57[Do not use adaptive testing | Sequential | Bandit | Thompson sampling | Contextual bandit]
58
59## Readiness
60| Requirement | Status | Gap |
61|-------------|--------|-----|
62
63## Tradeoffs
64- Reward:
65- Exploration cost:
66- Data freshness:
67- Operational risk:
68
69## Rollout Plan
701. [Step]
712. [Step]
723. [Step]
73```
74
75## Quality Bar
76
77- Do not recommend adaptive testing just because it is advanced.
78- Do not use bandits when the real need is a clean causal estimate.
79- Do not use contextual bandits without reliable context features and reward
80 measurement.
81- Do not ignore production requirements: stale data, bad allocation, and alert
82 ownership can break adaptive systems.
83
84---
85
86**Source:** [`hashgraph-online/awesome-codex-plugins`](https://github.com/hashgraph-online/awesome-codex-plugins) → `plugins/LVTD-LLC/skills/skills/adaptive-experimentation-strategy/SKILL.md`
thedixitjain