A/B Test Setup
When to Use
Use this skill when you need when the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this,"...
You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
Initial Assessment
Check for product marketing context first:
If .agents/product-marketing.md exists (or .claude/product-marketing.md, or the legacy product-marketing-context.md filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
Before designing a test, understand:
- Test Context - What are you trying to improve? What change are you considering?
- Current State - Baseline conversion rate? Current traffic volume?
- Constraints - Technical complexity? Timeline? Tools available?
Core Principles
1. Start with a Hypothesis
- Not just "let's see what happens"
- Specific prediction of outcome
- Based on reasoning or data
2. Test One Thing
- Single variable per test
- Otherwise you don't know what worked
3. Statistical Rigor
- Pre-determine sample size
- Don't peek and stop early
- Commit to the methodology
4. Measure What Matters
- Primary metric tied to business value
- Secondary metrics for context
- Guardrail metrics to prevent harm
Hypothesis Framework
Structure
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
Example
Weak: "Changing the button color might increase clicks."
Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
Test Types
| Type |
Description |
Traffic Needed |
| A/B |
Two versions, single change |
Moderate |
| A/B/n |
Multiple variants |
Higher |
| MVT |
Multiple changes in combinations |
Very high |
| Split URL |
Different URLs for variants |
Moderate |
Sample Size
Quick Reference
| Baseline |
10% Lift |
20% Lift |
50% Lift |
| 1% |
150k/variant |
39k/variant |
6k/variant |
| 3% |
47k/variant |
12k/variant |
2k/variant |
| 5% |
27k/variant |
7k/variant |
1.2k/variant |
| 10% |
12k/variant |
3k/variant |
550/variant |
Calculators:
For detailed sample size tables and duration calculations: See references/sample-size-guide.md
Metrics Selection
Primary Metric
- Single metric that matters most
- Directly tied to hypothesis
- What you'll use to call the test
Secondary Metrics
- Support primary metric interpretation
- Explain why/how the change worked
Guardrail Metrics
- Things that shouldn't get worse
- Stop test if significantly negative
Example: Pricing Page Test
- Primary: Plan selection rate
- Secondary: Time on page, plan distribution
- Guardrail: Support tickets, refund rate
Designing Variants
What to Vary
| Category |
Examples |
| Headlines/Copy |
Message angle, value prop, specificity, tone |
| Visual Design |
Layout, color, images, hierarchy |
| CTA |
Button copy, size, placement, number |
| Content |
Information included, order, amount, social proof |
Best Practices
- Single, meaningful change
- Bold enough to make a difference
- True to the hypothesis
Traffic Allocation
| Approach |
Split |
When to Use |
| Standard |
50/50 |
Default for A/B |
| Conservative |
90/10, 80/20 |
Limit risk of bad variant |
| Ramping |
Start small, increase |
Technical risk mitigation |
Considerations:
- Consistency: Users see same variant on return
- Balanced exposure across time of day/week
Implementation
Client-Side
- JavaScript modifies page after load
- Quick to implement, can cause flicker
- Tools: PostHog, Optimizely, VWO
Server-Side
- Variant determined before render
- No flicker, requires dev work
- Tools: PostHog, LaunchDarkly, Split
Running the Test
Pre-Launch Checklist
During the Test
DO:
- Monitor for technical issues
- Check segment quality
- Document external factors
**Avoi
1---2name: ab-testing3description: When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. 4category: Document Processing5source: antigravity6tags: [javascript, claude, ai, agent, template, design, document, image, security, rag]7url: https://github.com/sickn33/antigravity-awesome-skills/tree/main/skills/ab-testing8metadata:9 source: https://skillmd.com/skills/sickn33/ab-testing10---111213# A/B Test Setup14## When to Use1516Use this skill when you need when the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this,"...171819You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.2021## Initial Assessment2223**Check for product marketing context first:**24If `.agents/product-marketing.md` exists (or `.claude/product-marketing.md`, or the legacy `product-marketing-context.md` filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.2526Before designing a test, understand:27281. **Test Context** - What are you trying to improve? What change are you considering?292. **Current State** - Baseline conversion rate? Current traffic volume?303. **Constraints** - Technical complexity? Timeline? Tools available?3132---3334## Core Principles3536### 1. Start with a Hypothesis37- Not just "let's see what happens"38- Specific prediction of outcome39- Based on reasoning or data4041### 2. Test One Thing42- Single variable per test43- Otherwise you don't know what worked4445### 3. Statistical Rigor46- Pre-determine sample size47- Don't peek and stop early48- Commit to the methodology4950### 4. Measure What Matters51- Primary metric tied to business value52- Secondary metrics for context53- Guardrail metrics to prevent harm5455---5657## Hypothesis Framework5859### Structure6061```62Because [observation/data],63we believe [change]64will cause [expected outcome]65for [audience].66We'll know this is true when [metrics].67```6869### Example7071**Weak**: "Changing the button color might increase clicks."7273**Strong**: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."7475---7677## Test Types7879| Type | Description | Traffic Needed |80|------|-------------|----------------|81| A/B | Two versions, single change | Moderate |82| A/B/n | Multiple variants | Higher |83| MVT | Multiple changes in combinations | Very high |84| Split URL | Different URLs for variants | Moderate |8586---8788## Sample Size8990### Quick Reference9192| Baseline | 10% Lift | 20% Lift | 50% Lift |93|----------|----------|----------|----------|94| 1% | 150k/variant | 39k/variant | 6k/variant |95| 3% | 47k/variant | 12k/variant | 2k/variant |96| 5% | 27k/variant | 7k/variant | 1.2k/variant |97| 10% | 12k/variant | 3k/variant | 550/variant |9899**Calculators:**100- [Evan Miller's](https://www.evanmiller.org/ab-testing/sample-size.html)101- [Optimizely's](https://www.optimizely.com/sample-size-calculator/)102103**For detailed sample size tables and duration calculations**: See [references/sample-size-guide.md](references/sample-size-guide.md)104105---106107## Metrics Selection108109### Primary Metric110- Single metric that matters most111- Directly tied to hypothesis112- What you'll use to call the test113114### Secondary Metrics115- Support primary metric interpretation116- Explain why/how the change worked117118### Guardrail Metrics119- Things that shouldn't get worse120- Stop test if significantly negative121122### Example: Pricing Page Test123- **Primary**: Plan selection rate124- **Secondary**: Time on page, plan distribution125- **Guardrail**: Support tickets, refund rate126127---128129## Designing Variants130131### What to Vary132133| Category | Examples |134|----------|----------|135| Headlines/Copy | Message angle, value prop, specificity, tone |136| Visual Design | Layout, color, images, hierarchy |137| CTA | Button copy, size, placement, number |138| Content | Information included, order, amount, social proof |139140### Best Practices141- Single, meaningful change142- Bold enough to make a difference143- True to the hypothesis144145---146147## Traffic Allocation148149| Approach | Split | When to Use |150|----------|-------|-------------|151| Standard | 50/50 | Default for A/B |152| Conservative | 90/10, 80/20 | Limit risk of bad variant |153| Ramping | Start small, increase | Technical risk mitigation |154155**Considerations:**156- Consistency: Users see same variant on return157- Balanced exposure across time of day/week158159---160161## Implementation162163### Client-Side164- JavaScript modifies page after load165- Quick to implement, can cause flicker166- Tools: PostHog, Optimizely, VWO167168### Server-Side169- Variant determined before render170- No flicker, requires dev work171- Tools: PostHog, LaunchDarkly, Split172173---174175## Running the Test176177### Pre-Launch Checklist178- [ ] Hypothesis documented179- [ ] Primary metric defined180- [ ] Sample size calculated181- [ ] Variants implemented correctly182- [ ] Tracking verified183- [ ] QA completed on all variants184185### During the Test186187**DO:**188- Monitor for technical issues189- Check segment quality190- Document external factors191192**Avoi