A/B Testing Platform Strategy
Use this skill to decide how an organization should support A/B testing through
platform choices, architecture, ownership, and incremental scope.
Source Traceability
Primary source: Practical A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from chapter 5 lines 3804-4443. Related startup
and "start simple" context comes from preface lines 286-332 and chapter 2 lines
1622-1628.
Related Advanced Skills
experimentation-strategy-roadmap: use when deciding which platform
capability to prioritize across rate, quality, cost, usability, and company
strategy.experimentation-throughput-strategy: use when the platform needs capacity
visibility, isolated versus overlapping test policies, or coordination tools.experiment-verification-monitoring: use when the platform needs QA tooling,
canaries, A/A tests, active monitoring, leakage checks, or quality metrics.adaptive-experimentation-strategy: use when considering sequential testing,
bandits, Thompson sampling, contextual bandits, or dynamic allocation support.
Reference Routing
| Need |
Read |
| Platform concepts and components |
references/core/knowledge.md |
| Build-vs-buy and scoping rules |
references/core/rules.md |
| Scenario examples |
references/core/examples.md |
| Decision workflow |
workflows/decide-platform-strategy.md |
Workflow
- State the experimentation goals and current constraints.
- Inventory required platform components.
- Separate must-have launch capability from later platform maturity.
- Compare build, buy, and hybrid options against team capacity and risk.
- Plan data, assignment, exposure logging, metrics, and reporting ownership.
- Define the smallest useful platform and the triggers for expanding it.
Output Format
# A/B Testing Platform Strategy
## Recommendation
[Build | Buy | Hybrid | Start manually] because [reason].
## Current Context
- Team:
- Product surface:
- Experiment volume:
- Data maturity:
- Engineering capacity:
## Required Capabilities
| Capability | Need Now? | Build/Buy/Manual | Owner |
|------------|-----------|------------------|-------|
## Tradeoffs
- Build advantages:
- Build risks:
- Buy advantages:
- Buy risks:
- Hybrid notes:
## Incremental Roadmap
1. Minimum viable experimentation:
2. Reliability and governance:
3. Scale and self-service:
Quality Bar
- Do not recommend building a full platform before the team has proven demand.
- Do not recommend buying without checking integration, data, and governance
fit.
- Keep data and exposure logging first-class; a platform without trustworthy
measurement creates false confidence.
- Treat platform scope as evolutionary, not all-or-nothing.
1---2name: ab-testing-platform-strategy3description: Plan A/B testing platform strategy, architecture, and build-vs-buy decisions for product engineering teams. Use when deciding whether to build or buy an experimentation platform, scoping feature flagging, targeting, assignment, exposure logging, metrics pipelines, dashboards, governance, or evolving a simple testing setup into a durable platform.4license: MIT5---67# A/B Testing Platform Strategy89Use this skill to decide how an organization should support A/B testing through10platform choices, architecture, ownership, and incremental scope.1112## Source Traceability1314Primary source: Practical A/B Testing by Leemay Nassery. Guidance is15transformed and paraphrased from chapter 5 lines 3804-4443. Related startup16and "start simple" context comes from preface lines 286-332 and chapter 2 lines171622-1628.1819## Related Advanced Skills2021- `experimentation-strategy-roadmap`: use when deciding which platform22 capability to prioritize across rate, quality, cost, usability, and company23 strategy.24- `experimentation-throughput-strategy`: use when the platform needs capacity25 visibility, isolated versus overlapping test policies, or coordination tools.26- `experiment-verification-monitoring`: use when the platform needs QA tooling,27 canaries, A/A tests, active monitoring, leakage checks, or quality metrics.28- `adaptive-experimentation-strategy`: use when considering sequential testing,29 bandits, Thompson sampling, contextual bandits, or dynamic allocation support.3031## Reference Routing3233| Need | Read |34|------|------|35| Platform concepts and components | `references/core/knowledge.md` |36| Build-vs-buy and scoping rules | `references/core/rules.md` |37| Scenario examples | `references/core/examples.md` |38| Decision workflow | `workflows/decide-platform-strategy.md` |3940## Workflow41421. State the experimentation goals and current constraints.432. Inventory required platform components.443. Separate must-have launch capability from later platform maturity.454. Compare build, buy, and hybrid options against team capacity and risk.465. Plan data, assignment, exposure logging, metrics, and reporting ownership.476. Define the smallest useful platform and the triggers for expanding it.4849## Output Format5051```markdown52# A/B Testing Platform Strategy5354## Recommendation55[Build | Buy | Hybrid | Start manually] because [reason].5657## Current Context58- Team:59- Product surface:60- Experiment volume:61- Data maturity:62- Engineering capacity:6364## Required Capabilities65| Capability | Need Now? | Build/Buy/Manual | Owner |66|------------|-----------|------------------|-------|6768## Tradeoffs69- Build advantages:70- Build risks:71- Buy advantages:72- Buy risks:73- Hybrid notes:7475## Incremental Roadmap761. Minimum viable experimentation:772. Reliability and governance:783. Scale and self-service:79```8081## Quality Bar8283- Do not recommend building a full platform before the team has proven demand.84- Do not recommend buying without checking integration, data, and governance85 fit.86- Keep data and exposure logging first-class; a platform without trustworthy87 measurement creates false confidence.88- Treat platform scope as evolutionary, not all-or-nothing.
LVTD-LLC