A/B Test Setup - Experimentation Design & Analysis
Category: Product Team
Tags: A/B testing, experiments, statistical significance, sample size, feature flags, hypothesis testing
Overview
A/B Test Setup provides the complete framework for designing experiments that produce statistically valid, actionable results. Most A/B tests fail not because the variant was wrong, but because the test was poorly designed: wrong sample size, wrong metric, or someone peeked at results and stopped early. This skill prevents those mistakes.
Clarify First
Before designing the experiment, confirm these inputs. If any is unknown or vague, ASK — do not assume:
Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.
The Experiment Lifecycle
1. HYPOTHESIZE → 2. DESIGN → 3. CALCULATE → 4. IMPLEMENT
↑ │
│ ▼
7. ITERATE ← 6. DOCUMENT ← 5. ANALYZE ← [Run to completion]
Step 1: Hypothesis Formulation
The Hypothesis Template
Because [observation or data point],
we believe [specific change]
will cause [measurable outcome]
for [defined audience segment].
We'll know this is true when [primary metric] changes by [minimum detectable effect].
We'll watch [guardrail metrics] to ensure no negative impact.
Good vs Bad Hypotheses
| Quality |
Hypothesis |
Problem |
| Bad |
"Changing the button color might increase clicks" |
No data basis, no target, no measurement plan |
| Mediocre |
"A green button will get more clicks than blue" |
No "why", no target size, no guardrails |
| Good |
"Because heatmaps show 40% of users don't notice our CTA, making the button 2x larger with contrasting color will increase CTA clicks by 15%+ for new visitors. Guardrail: page load time stays under 2s." |
Data-backed, specific change, measurable outcome, defined audience, guardrail |
Hypothesis Sources (Where to Find Test Ideas)
| Source |
What to Look For |
Example |
| Analytics data |
Drop-off points, low-performing pages |
"80% of users drop off at step 3 of onboarding" |
| User research |
Confusion, frustration, unmet needs |
"Users don't understand what the product does from the homepage" |
| Heatmaps/session recordings |
Ignored elements, rage clicks |
"Nobody scrolls past the fold on pricing page" |
| Support tickets |
Recurring complaints, feature confusion |
"Users constantly ask how to invite team members" |
| Competitor analysis |
Different approaches to same problem |
"Competitor uses a wizard; we use a form" |
| Sales objections |
Common reasons prospects don't convert |
"Prospects want to see pricing before signing up" |
Step 2: Test Design
Test Types
| Type |
Variants |
Traffic Need |
Best For |
| A/B |
2 (control + 1 variant) |
Moderate |
Single change validation |
| A/B/n |
3+ variants |
High |
Comparing multiple approaches |
| Multivariate (MVT) |
Combinations of changes |
Very high |
Optimizing multiple elements |
| Split URL |
Different pages |
Moderate |
Major redesigns |
| Bandit |
Dynamic allocation |
Low-moderate |
Revenue optimization |
Default recommendation: Standard A/B test. Only use A/B/n or MVT when you have enough traffic and a specific need.
What to Test (By Impact)
| Category |
High Impact |
Medium Impact |
Low Impact |
| Copy |
Headline/value prop, CTA text |
Body copy, social proof |
Microcopy, labels |
| Design |
Page layout, above-fold content |
Visual hierarchy, imagery |
Color, font size |
| UX |
Number of steps, form fields |
Button placement, navigation |
Animations, transitions |
| Pricing |
Price point, plan names |
Feature packaging, anchoring |
Billing frequency display |
| Social Proof |
Testimonials vs none, logos |
Testimonial format, placement |
Testimonial count |
Metric Selection
Every test needs three types of metrics:
Primary Metric (1 only)
- The single metric that determines success
- Directly tied to the hypothesis
- Must be measurable within the test duration
- Examples: signup rate, click-through rate, purchase rate
Secondary Metrics (2-3)
- Explain why the primary metric moved
- Provide context for decision-making
- Examples: time on page, scroll depth, feature adoption rate
Guardrail Metrics (1-3)
- Things that must NOT get worse
- Stop the test if significantly negative
- Examples: error rate, support ticket volume, page load time, refund rate
Step 3: Sample Size Calculation
Quick Reference Table
Minimum visitors PER VARIANT needed (95% confidence, 80% power):
| Baseline Rate |
5% Lift |
10% Lift |
15% Lift |
20% Lift |
50% Lift |
| 1% |
620,000 |
156,000 |
70,000 |
39,000 |
6,400 |
| 2% |
305,000 |
77,000 |
34,000 |
19,500 |
3,200 |
| 3% |
200,000 |
51,000 |
23,000 |
12,800 |
2,100 |
| 5% |
116,000 |
29,500 |
13,200 |
7,500 |
1,250 |
| 10% |
54,000 |
13,800 |
6,200 |
3,500 |
600 |
| 20% |
24,000 |
6,200 |
2,800 |
1,600 |
280 |
| 50% |
6,100 |
1,600 |
720 |
410 |
75 |
Duration Calculation
Duration (days) = (Sample size per variant * Number of variants) / Daily traffic to test page
Minimum duration: 7 days (to capture day-of-week effects)
Maximum recommended: 6 weeks (beyond this, external factors contaminate results)
What If You Don't Have Enough Traffic?
| Situation |
Solution |
| Need 100K visitors, get 5K/week |
Increase minimum detectable effect (test bolder changes) |
| Very low traffic (<1K/week) |
Use qualitative testing (user testing, surveys) instead |
| Medium traffic (5-20K/week) |
Run for 4-6 weeks, test big changes only |
| High traffic (50K+/week) |
You can test subtle changes, run multiple tests |
Step 4: Implementation
Client-Side Implementation
JavaScript modifies the page after initial render.
Pros: Quick to implement, no deploy needed
Cons: Can cause flicker (flash of original content), blocked by ad blockers
Tools: PostHog, Optimizely, VWO, Google Optimize
Anti-flicker pattern:
// Add to <head> before any rendering
<style>.ab-test-hide { opacity: 0 !important; }</style>
<script>document.documentElement.classList.add('ab-test-hide');</script>
// In your test script (runs after variant assignment):
document.documentElement.classList.remove('ab-test-hide');
Server-Side Implementation
Variant determined before page renders. No flicker, no client-side dependency.
Pros: No flicker, not blocked by ad blockers, works for logged-in features
Cons: Requires engineering work, deploy needed
Tools: PostHog, LaunchDarkly, Split, Unleash, custom feature flags
Basic feature flag pattern:
# Server-side variant assignment
def get_variant(user_id: str, experiment: str) -> str:
# Deterministic hash ensures same user always sees same variant
hash_input = f"{user_id}:{experiment}"
hash_value = hashlib.md5(hash_input.encode()).hexdigest()
bucket = int(hash_value[:8], 16) % 100
if bucket < 50:
return "control"
else:
return "variant"
Traffic Allocation
| Strategy |
Split |
When to Use |
| Standard |
50/50 |
Default. Maximum statistical power. |
| Conservative |
90/10 or 80/20 |
Risky changes, revenue-impacting tests |
| Ramped |
Start 95/5, increase to 50/50 |
New infrastructure, technical risk |
Critical rules:
- Users must see the same variant on every visit (sticky assignment by user ID or cookie)
- Allocation must be balanced across time of day and day of week
- Never change allocation mid-test
Step 5: Running the Test
Pre-Launch Checklist
During the Test
DO:
- Monitor for technical errors (variant not rendering, tracking broken)
- Check that traffic split is balanced daily
- Document any external events that might affect results
DO NOT:
- Look at results before reaching sample size ("peeking problem")
- Make changes to either variant
- Add traffic from new sources mid-test
- Stop the test early because one variant "looks like it's winning"
The Peeking Problem (Critical)
Looking at results before reaching the planned sample size and stopping because one variant looks better leads to a 25-40% false positive rate (vs the intended 5%).
Why: Statistical significance fluctuates wildly with small samples. A variant can show p < 0.05 at 20% of planned sample size and p > 0.30 at full sample.
Solutions:
- Pre-commit to sample size and do not check results until reached
- If you must monitor: use sequential testing methods (group sequential design, always-valid p-values)
- Set calendar reminder for expected completion date -- that is when you look
Step 6: Analysis
Analysis Checklist
- Did we reach planned sample size? If not, results are preliminary only.
- Is it statistically significant? p < 0.05 = 95% confidence the difference is real.
- What's the confidence interval? Tells you the range of likely true effect.
- Is the effect size meaningful? A 0.1% lift that's "significant" may not be worth implementing.
- Are secondary metrics consistent? Do they support the primary result?
- Any guardrail violations? Did anything get worse?
- Segment analysis: Different results for mobile vs desktop? New vs returning?
Interpreting Results
| Result |
Primary Metric |
Confidence |
Action |
| Clear winner |
Variant +15%, p < 0.01 |
High |
Implement variant |
| Modest winner |
Variant +5%, p < 0.05 |
Medium |
Implement if easy, else run longer |
| Flat |
< 2% difference, p > 0.20 |
High (no effect) |
Keep control, test something bolder |
| Loser |
Variant -10%, p < 0.05 |
High |
Keep control, investigate why |
| Inconclusive |
5% difference, p = 0.08 |
Low |
Need more traffic or bolder test |
| Mixed signals |
Primary up, guardrail down |
Investigate |
Dig into segments, do not ship blindly |
Common Analysis Mistakes
| Mistake |
Consequence |
Prevention |
| Stopping at first significance |
25-40% false positive rate |
Commit to sample size |
| Cherry-picking segments |
Finding "winners" that don't replicate |
Pre-register segments of interest |
| Ignoring confidence intervals |
Overestimating effect size |
Always report CI alongside p-value |
| Multiple comparisons |
Inflated Type I error |
Bonferroni correction for A/B/n |
| Survivorship bias |
Only analyzing users who completed flow |
Include all users from assignment point |
| Simpson's paradox |
Aggregate hides segment reversal |
Always check key segments |
Step 7: Documentation
Every test must be documented, regardless of outcome.
Test Documentation Template
EXPERIMENT: [Name]
DATE: [Start] to [End]
OWNER: [Name]
HYPOTHESIS:
Because [observation], we believed [change] would cause [outcome] for [audience].
VARIANTS:
- Control: [description]
- Variant: [description + screenshot]
METRICS:
- Primary: [metric] (baseline: [X]%, MDE: [Y]%)
- Secondary: [metrics]
- Guardrails: [metrics]
RESULTS:
- Sample size: [actual] / [planned]
- Duration: [X] days
- Primary metric: Control [X]% vs Variant [Y]% (p = [Z], CI: [range])
- Secondary metrics: [results]
- Guardrails: [all clear / violation noted]
DECISION: [Ship variant / Keep control / Iterate]
LEARNINGS:
- [What we learned about our users]
- [What we'd do differently next time]
Experiment Prioritization Framework
ICE Scoring
| Factor |
Score (1-10) |
Question |
| Impact |
How much will this move the metric? |
Big change to primary KPI = 10 |
| Confidence |
How sure are we it will work? |
Strong data supporting hypothesis = 10 |
| Ease |
How easy is it to implement and measure? |
Can ship in a day = 10 |
ICE Score = (Impact + Confidence + Ease) / 3
Rank all test ideas by ICE score. Run highest first.
Test Backlog Template
| # |
Hypothesis |
Primary Metric |
ICE |
Est. Duration |
Status |
| 1 |
Larger CTA increases signups |
Signup rate |
8.3 |
2 weeks |
Ready |
| 2 |
Social proof on pricing increases conversion |
Plan selection rate |
7.0 |
3 weeks |
Needs design |
| 3 |
Shorter onboarding increases activation |
Feature activation |
6.7 |
4 weeks |
In backlog |
Proactive Triggers
- Someone debates between two design options: propose an A/B test instead of opinionating
- Conversion rate mentioned as underperforming: offer to design a test, not guess at solutions
- Pricing page changes discussed: always test pricing changes with guardrail metrics
- Post-launch of any feature: propose follow-up experiment to optimize
- "Let's just try it and see": redirect to structured hypothesis before implementation
Related Skills
| Skill |
Use When |
| analytics-tracking |
Setting up event tracking that feeds experiment metrics |
| campaign-analytics |
Folding experiment results into broader attribution |
| launch-strategy |
Testing within a product launch sequence |
| prompt-engineer-toolkit |
A/B testing AI prompts in production |
Tool Reference
sample_size_calculator.py
Calculates required sample size per variant using the normal approximation to the two-proportion z-test. Includes Bonferroni correction for multi-variant tests and duration estimation.
| Flag |
Type |
Default |
Description |
--baseline, -b |
float |
(required) |
Baseline conversion rate (e.g. 0.05 for 5%) |
--mde, -m |
float |
(required) |
Minimum detectable effect as relative lift (e.g. 0.10 for 10%) |
--alpha, -a |
float |
0.05 |
Significance level |
--power, -p |
float |
0.80 |
Statistical power |
--variants, -v |
int |
2 |
Number of variants including control |
--daily-traffic, -d |
int |
0 |
Daily eligible traffic for duration estimation |
--one-tailed |
flag |
False |
Use one-tailed test instead of two-tailed |
--json |
flag |
False |
Output as JSON |
python scripts/sample_size_calculator.py --baseline 0.05 --mde 0.10
python scripts/sample_size_calculator.py --baseline 0.12 --mde 0.15 --power 0.9 --daily-traffic 5000
python scripts/sample_size_calculator.py --baseline 0.05 --mde 0.10 --variants 3 --json
experiment_analyzer.py
Analyzes A/B test results using the two-proportion z-test with confidence intervals and segment breakdown.
| Flag |
Type |
Default |
Description |
input |
positional |
(required) |
CSV file with results or "sample" to create sample |
--alpha, -a |
float |
0.05 |
Significance level |
--json |
flag |
False |
Output as JSON |
CSV format: variant,visitors,conversions,segment
python scripts/experiment_analyzer.py sample
python scripts/experiment_analyzer.py results.csv
python scripts/experiment_analyzer.py results.csv --alpha 0.01 --json
experiment_planner.py
Generates a structured experiment plan from a hypothesis text, including metric selection, sample size, timeline, risks, and documentation template.
| Flag |
Type |
Default |
Description |
--hypothesis, -H |
string |
(required) |
Experiment hypothesis text |
--baseline, -b |
float |
0.05 |
Baseline conversion rate |
--mde, -m |
float |
0.10 |
Minimum detectable effect as relative lift |
--daily-traffic, -d |
int |
0 |
Daily eligible traffic |
--variants, -v |
int |
2 |
Number of variants including control |
--json |
flag |
False |
Output as JSON |
python scripts/experiment_planner.py --hypothesis "Larger CTA will increase signups by 15%"
python scripts/experiment_planner.py -H "Simplified checkout boosts conversions" -b 0.08 -m 0.15 -d 3000
python scripts/experiment_planner.py -H "New pricing page" --json
Troubleshooting
| Problem |
Cause |
Solution |
| Sample size is unrealistically large |
MDE too small or baseline too low |
Increase MDE (test bolder changes) or target a higher-traffic page |
| Test duration exceeds 6 weeks |
Insufficient daily traffic |
Consider qualitative methods, test bigger changes, or combine traffic from multiple pages |
| p-value hovers around 0.05 |
Borderline significance |
Do not stop early; run to planned sample size or extend 20% |
| Results significant but lift is tiny (<1%) |
Overpowered test |
Check practical significance alongside statistical significance |
| Segment results contradict overall |
Simpson's paradox |
Investigate segment composition; report both overall and segment results |
| Variant performs differently on mobile vs desktop |
Device-specific UX issues |
Design device-specific variants; increase per-segment sample size |
| Calculator produces negative CI |
Very small samples or extreme rates |
Ensure sufficient sample size; check data integrity |
Success Criteria
| Criterion |
Target |
How to Measure |
| Tests reach planned sample size |
100% of tests |
Compare actual vs planned sample at conclusion |
| False positive rate |
<5% |
Track post-implementation lift vs test prediction |
| Test velocity |
2+ tests per team per month |
Count experiments documented per sprint |
| Documentation completeness |
100% of tests documented |
Audit experiment records quarterly |
| Average test duration |
<4 weeks |
Measure start-to-conclusion calendar days |
| Decision quality |
>80% of shipped variants hold gains at 90 days |
Post-ship metric tracking |
Scope & Limitations
In scope:
- Hypothesis formulation and validation
- Sample size and power calculations
- Frequentist two-proportion z-tests
- A/B, A/B/n, and split URL test planning
- Segment-level analysis
- Pre/post test documentation
Out of scope:
- Bayesian A/B testing methods (use dedicated Bayesian tools)
- Multi-armed bandit algorithms (require real-time allocation infrastructure)
- Multivariate testing (MVT) analysis (combinatorial explosion requires specialized tools)
- Server-side feature flag implementation (see engineering skills)
- Revenue-based metrics requiring transaction-level data
- Sequential testing / always-valid p-values (use Optimizely Stats Engine or similar)
Integration Points
| Tool / Platform |
Integration Method |
Use Case |
| PostHog / Amplitude |
JSON export from experiment_analyzer |
Feed results into product analytics |
| Jira / Linear |
experiment_planner JSON output |
Create experiment tickets with metadata |
| Google Sheets |
CSV export from experiment_analyzer |
Share results with non-technical stakeholders |
| LaunchDarkly / Unleash |
experiment_planner checklist |
Pre-launch validation before feature flag rollout |
| Slack / Notion |
Copy human-readable output |
Async experiment status updates |
| CI/CD pipelines |
--json flag on all scripts |
Automated experiment health checks |
1---2name: ab-test-setup-23description: Design and run statistically rigorous A/B tests and experiments. Use when planning experiments, calculating sample sizes, designing test variants, selecting metrics, analyzing results, or when someone says "let's test that."4license: MIT + Commons Clause5---6# A/B Test Setup - Experimentation Design & Analysis78**Category:** Product Team9**Tags:** A/B testing, experiments, statistical significance, sample size, feature flags, hypothesis testing1011## Overview1213A/B Test Setup provides the complete framework for designing experiments that produce statistically valid, actionable results. Most A/B tests fail not because the variant was wrong, but because the test was poorly designed: wrong sample size, wrong metric, or someone peeked at results and stopped early. This skill prevents those mistakes.1415---1617## Clarify First1819Before designing the experiment, confirm these inputs. If any is unknown or vague, ASK — do not assume:2021- [ ] **Primary metric + minimum detectable effect** — the one success metric and smallest lift worth detecting (drives sample size, duration, and metric selection)22- [ ] **Baseline conversion rate** — current rate for the primary metric (sets required sample size per variant)23- [ ] **Available traffic to the test surface** — daily eligible visitors (decides whether the test is feasible or needs a bolder change / qualitative method)24- [ ] **The change and its rationale** — what varies and the data behind it (drives the hypothesis and variant design)2526Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.2728## The Experiment Lifecycle2930```311. HYPOTHESIZE → 2. DESIGN → 3. CALCULATE → 4. IMPLEMENT32 ↑ │33 │ ▼347. ITERATE ← 6. DOCUMENT ← 5. ANALYZE ← [Run to completion]35```3637---3839## Step 1: Hypothesis Formulation4041### The Hypothesis Template4243```44Because [observation or data point],45we believe [specific change]46will cause [measurable outcome]47for [defined audience segment].4849We'll know this is true when [primary metric] changes by [minimum detectable effect].50We'll watch [guardrail metrics] to ensure no negative impact.51```5253### Good vs Bad Hypotheses5455| Quality | Hypothesis | Problem |56|---------|-----------|---------|57| Bad | "Changing the button color might increase clicks" | No data basis, no target, no measurement plan |58| Mediocre | "A green button will get more clicks than blue" | No "why", no target size, no guardrails |59| Good | "Because heatmaps show 40% of users don't notice our CTA, making the button 2x larger with contrasting color will increase CTA clicks by 15%+ for new visitors. Guardrail: page load time stays under 2s." | Data-backed, specific change, measurable outcome, defined audience, guardrail |6061### Hypothesis Sources (Where to Find Test Ideas)6263| Source | What to Look For | Example |64|--------|-----------------|---------|65| Analytics data | Drop-off points, low-performing pages | "80% of users drop off at step 3 of onboarding" |66| User research | Confusion, frustration, unmet needs | "Users don't understand what the product does from the homepage" |67| Heatmaps/session recordings | Ignored elements, rage clicks | "Nobody scrolls past the fold on pricing page" |68| Support tickets | Recurring complaints, feature confusion | "Users constantly ask how to invite team members" |69| Competitor analysis | Different approaches to same problem | "Competitor uses a wizard; we use a form" |70| Sales objections | Common reasons prospects don't convert | "Prospects want to see pricing before signing up" |7172---7374## Step 2: Test Design7576### Test Types7778| Type | Variants | Traffic Need | Best For |79|------|----------|-------------|---------|80| A/B | 2 (control + 1 variant) | Moderate | Single change validation |81| A/B/n | 3+ variants | High | Comparing multiple approaches |82| Multivariate (MVT) | Combinations of changes | Very high | Optimizing multiple elements |83| Split URL | Different pages | Moderate | Major redesigns |84| Bandit | Dynamic allocation | Low-moderate | Revenue optimization |8586**Default recommendation:** Standard A/B test. Only use A/B/n or MVT when you have enough traffic and a specific need.8788### What to Test (By Impact)8990| Category | High Impact | Medium Impact | Low Impact |91|----------|-----------|---------------|-----------|92| **Copy** | Headline/value prop, CTA text | Body copy, social proof | Microcopy, labels |93| **Design** | Page layout, above-fold content | Visual hierarchy, imagery | Color, font size |94| **UX** | Number of steps, form fields | Button placement, navigation | Animations, transitions |95| **Pricing** | Price point, plan names | Feature packaging, anchoring | Billing frequency display |96| **Social Proof** | Testimonials vs none, logos | Testimonial format, placement | Testimonial count |9798### Metric Selection99100Every test needs three types of metrics:101102**Primary Metric (1 only)**103- The single metric that determines success104- Directly tied to the hypothesis105- Must be measurable within the test duration106- Examples: signup rate, click-through rate, purchase rate107108**Secondary Metrics (2-3)**109- Explain why the primary metric moved110- Provide context for decision-making111- Examples: time on page, scroll depth, feature adoption rate112113**Guardrail Metrics (1-3)**114- Things that must NOT get worse115- Stop the test if significantly negative116- Examples: error rate, support ticket volume, page load time, refund rate117118---119120## Step 3: Sample Size Calculation121122### Quick Reference Table123124Minimum visitors PER VARIANT needed (95% confidence, 80% power):125126| Baseline Rate | 5% Lift | 10% Lift | 15% Lift | 20% Lift | 50% Lift |127|--------------|---------|----------|----------|----------|----------|128| 1% | 620,000 | 156,000 | 70,000 | 39,000 | 6,400 |129| 2% | 305,000 | 77,000 | 34,000 | 19,500 | 3,200 |130| 3% | 200,000 | 51,000 | 23,000 | 12,800 | 2,100 |131| 5% | 116,000 | 29,500 | 13,200 | 7,500 | 1,250 |132| 10% | 54,000 | 13,800 | 6,200 | 3,500 | 600 |133| 20% | 24,000 | 6,200 | 2,800 | 1,600 | 280 |134| 50% | 6,100 | 1,600 | 720 | 410 | 75 |135136### Duration Calculation137138```139Duration (days) = (Sample size per variant * Number of variants) / Daily traffic to test page140```141142**Minimum duration:** 7 days (to capture day-of-week effects)143**Maximum recommended:** 6 weeks (beyond this, external factors contaminate results)144145### What If You Don't Have Enough Traffic?146147| Situation | Solution |148|-----------|----------|149| Need 100K visitors, get 5K/week | Increase minimum detectable effect (test bolder changes) |150| Very low traffic (<1K/week) | Use qualitative testing (user testing, surveys) instead |151| Medium traffic (5-20K/week) | Run for 4-6 weeks, test big changes only |152| High traffic (50K+/week) | You can test subtle changes, run multiple tests |153154---155156## Step 4: Implementation157158### Client-Side Implementation159160JavaScript modifies the page after initial render.161162**Pros:** Quick to implement, no deploy needed163**Cons:** Can cause flicker (flash of original content), blocked by ad blockers164**Tools:** PostHog, Optimizely, VWO, Google Optimize165166**Anti-flicker pattern:**167```javascript168// Add to <head> before any rendering169<style>.ab-test-hide { opacity: 0 !important; }</style>170<script>document.documentElement.classList.add('ab-test-hide');</script>171172// In your test script (runs after variant assignment):173document.documentElement.classList.remove('ab-test-hide');174```175176### Server-Side Implementation177178Variant determined before page renders. No flicker, no client-side dependency.179180**Pros:** No flicker, not blocked by ad blockers, works for logged-in features181**Cons:** Requires engineering work, deploy needed182**Tools:** PostHog, LaunchDarkly, Split, Unleash, custom feature flags183184**Basic feature flag pattern:**185```python186# Server-side variant assignment187def get_variant(user_id: str, experiment: str) -> str:188 # Deterministic hash ensures same user always sees same variant189 hash_input = f"{user_id}:{experiment}"190 hash_value = hashlib.md5(hash_input.encode()).hexdigest()191 bucket = int(hash_value[:8], 16) % 100192193 if bucket < 50:194 return "control"195 else:196 return "variant"197```198199### Traffic Allocation200201| Strategy | Split | When to Use |202|----------|-------|-------------|203| Standard | 50/50 | Default. Maximum statistical power. |204| Conservative | 90/10 or 80/20 | Risky changes, revenue-impacting tests |205| Ramped | Start 95/5, increase to 50/50 | New infrastructure, technical risk |206207**Critical rules:**208- Users must see the same variant on every visit (sticky assignment by user ID or cookie)209- Allocation must be balanced across time of day and day of week210- Never change allocation mid-test211212---213214## Step 5: Running the Test215216### Pre-Launch Checklist217218- [ ] Hypothesis documented with primary metric and minimum detectable effect219- [ ] Sample size calculated, expected duration estimated220- [ ] Both variants implemented and QA'd on all device types221- [ ] Tracking verified (events fire correctly for both variants)222- [ ] No other tests running on the same page/feature223- [ ] Stakeholders informed of test duration and "no peeking" rule224- [ ] External factor calendar checked (no major launches, holidays, press)225226### During the Test227228**DO:**229- Monitor for technical errors (variant not rendering, tracking broken)230- Check that traffic split is balanced daily231- Document any external events that might affect results232233**DO NOT:**234- Look at results before reaching sample size ("peeking problem")235- Make changes to either variant236- Add traffic from new sources mid-test237- Stop the test early because one variant "looks like it's winning"238239### The Peeking Problem (Critical)240241Looking at results before reaching the planned sample size and stopping because one variant looks better leads to a **25-40% false positive rate** (vs the intended 5%).242243Why: Statistical significance fluctuates wildly with small samples. A variant can show p < 0.05 at 20% of planned sample size and p > 0.30 at full sample.244245**Solutions:**2461. Pre-commit to sample size and do not check results until reached2472. If you must monitor: use sequential testing methods (group sequential design, always-valid p-values)2483. Set calendar reminder for expected completion date -- that is when you look249250---251252## Step 6: Analysis253254### Analysis Checklist2552561. **Did we reach planned sample size?** If not, results are preliminary only.2572. **Is it statistically significant?** p < 0.05 = 95% confidence the difference is real.2583. **What's the confidence interval?** Tells you the range of likely true effect.2594. **Is the effect size meaningful?** A 0.1% lift that's "significant" may not be worth implementing.2605. **Are secondary metrics consistent?** Do they support the primary result?2616. **Any guardrail violations?** Did anything get worse?2627. **Segment analysis:** Different results for mobile vs desktop? New vs returning?263264### Interpreting Results265266| Result | Primary Metric | Confidence | Action |267|--------|---------------|------------|--------|268| Clear winner | Variant +15%, p < 0.01 | High | Implement variant |269| Modest winner | Variant +5%, p < 0.05 | Medium | Implement if easy, else run longer |270| Flat | < 2% difference, p > 0.20 | High (no effect) | Keep control, test something bolder |271| Loser | Variant -10%, p < 0.05 | High | Keep control, investigate why |272| Inconclusive | 5% difference, p = 0.08 | Low | Need more traffic or bolder test |273| Mixed signals | Primary up, guardrail down | Investigate | Dig into segments, do not ship blindly |274275### Common Analysis Mistakes276277| Mistake | Consequence | Prevention |278|---------|-------------|------------|279| Stopping at first significance | 25-40% false positive rate | Commit to sample size |280| Cherry-picking segments | Finding "winners" that don't replicate | Pre-register segments of interest |281| Ignoring confidence intervals | Overestimating effect size | Always report CI alongside p-value |282| Multiple comparisons | Inflated Type I error | Bonferroni correction for A/B/n |283| Survivorship bias | Only analyzing users who completed flow | Include all users from assignment point |284| Simpson's paradox | Aggregate hides segment reversal | Always check key segments |285286---287288## Step 7: Documentation289290Every test must be documented, regardless of outcome.291292### Test Documentation Template293294```295EXPERIMENT: [Name]296DATE: [Start] to [End]297OWNER: [Name]298299HYPOTHESIS:300Because [observation], we believed [change] would cause [outcome] for [audience].301302VARIANTS:303- Control: [description]304- Variant: [description + screenshot]305306METRICS:307- Primary: [metric] (baseline: [X]%, MDE: [Y]%)308- Secondary: [metrics]309- Guardrails: [metrics]310311RESULTS:312- Sample size: [actual] / [planned]313- Duration: [X] days314- Primary metric: Control [X]% vs Variant [Y]% (p = [Z], CI: [range])315- Secondary metrics: [results]316- Guardrails: [all clear / violation noted]317318DECISION: [Ship variant / Keep control / Iterate]319320LEARNINGS:321- [What we learned about our users]322- [What we'd do differently next time]323```324325---326327## Experiment Prioritization Framework328329### ICE Scoring330331| Factor | Score (1-10) | Question |332|--------|-------------|----------|333| **Impact** | How much will this move the metric? | Big change to primary KPI = 10 |334| **Confidence** | How sure are we it will work? | Strong data supporting hypothesis = 10 |335| **Ease** | How easy is it to implement and measure? | Can ship in a day = 10 |336337**ICE Score = (Impact + Confidence + Ease) / 3**338339Rank all test ideas by ICE score. Run highest first.340341### Test Backlog Template342343| # | Hypothesis | Primary Metric | ICE | Est. Duration | Status |344|---|-----------|---------------|-----|---------------|--------|345| 1 | Larger CTA increases signups | Signup rate | 8.3 | 2 weeks | Ready |346| 2 | Social proof on pricing increases conversion | Plan selection rate | 7.0 | 3 weeks | Needs design |347| 3 | Shorter onboarding increases activation | Feature activation | 6.7 | 4 weeks | In backlog |348349---350351## Proactive Triggers352353- Someone debates between two design options: propose an A/B test instead of opinionating354- Conversion rate mentioned as underperforming: offer to design a test, not guess at solutions355- Pricing page changes discussed: always test pricing changes with guardrail metrics356- Post-launch of any feature: propose follow-up experiment to optimize357- "Let's just try it and see": redirect to structured hypothesis before implementation358359---360361## Related Skills362363| Skill | Use When |364|-------|----------|365| **analytics-tracking** | Setting up event tracking that feeds experiment metrics |366| **campaign-analytics** | Folding experiment results into broader attribution |367| **launch-strategy** | Testing within a product launch sequence |368| **prompt-engineer-toolkit** | A/B testing AI prompts in production |369370---371372## Tool Reference373374### sample_size_calculator.py375376Calculates required sample size per variant using the normal approximation to the two-proportion z-test. Includes Bonferroni correction for multi-variant tests and duration estimation.377378| Flag | Type | Default | Description |379|------|------|---------|-------------|380| `--baseline`, `-b` | float | (required) | Baseline conversion rate (e.g. 0.05 for 5%) |381| `--mde`, `-m` | float | (required) | Minimum detectable effect as relative lift (e.g. 0.10 for 10%) |382| `--alpha`, `-a` | float | 0.05 | Significance level |383| `--power`, `-p` | float | 0.80 | Statistical power |384| `--variants`, `-v` | int | 2 | Number of variants including control |385| `--daily-traffic`, `-d` | int | 0 | Daily eligible traffic for duration estimation |386| `--one-tailed` | flag | False | Use one-tailed test instead of two-tailed |387| `--json` | flag | False | Output as JSON |388389```bash390python scripts/sample_size_calculator.py --baseline 0.05 --mde 0.10391python scripts/sample_size_calculator.py --baseline 0.12 --mde 0.15 --power 0.9 --daily-traffic 5000392python scripts/sample_size_calculator.py --baseline 0.05 --mde 0.10 --variants 3 --json393```394395### experiment_analyzer.py396397Analyzes A/B test results using the two-proportion z-test with confidence intervals and segment breakdown.398399| Flag | Type | Default | Description |400|------|------|---------|-------------|401| `input` | positional | (required) | CSV file with results or "sample" to create sample |402| `--alpha`, `-a` | float | 0.05 | Significance level |403| `--json` | flag | False | Output as JSON |404405**CSV format:** `variant,visitors,conversions,segment`406407```bash408python scripts/experiment_analyzer.py sample409python scripts/experiment_analyzer.py results.csv410python scripts/experiment_analyzer.py results.csv --alpha 0.01 --json411```412413### experiment_planner.py414415Generates a structured experiment plan from a hypothesis text, including metric selection, sample size, timeline, risks, and documentation template.416417| Flag | Type | Default | Description |418|------|------|---------|-------------|419| `--hypothesis`, `-H` | string | (required) | Experiment hypothesis text |420| `--baseline`, `-b` | float | 0.05 | Baseline conversion rate |421| `--mde`, `-m` | float | 0.10 | Minimum detectable effect as relative lift |422| `--daily-traffic`, `-d` | int | 0 | Daily eligible traffic |423| `--variants`, `-v` | int | 2 | Number of variants including control |424| `--json` | flag | False | Output as JSON |425426```bash427python scripts/experiment_planner.py --hypothesis "Larger CTA will increase signups by 15%"428python scripts/experiment_planner.py -H "Simplified checkout boosts conversions" -b 0.08 -m 0.15 -d 3000429python scripts/experiment_planner.py -H "New pricing page" --json430```431432---433434## Troubleshooting435436| Problem | Cause | Solution |437|---------|-------|----------|438| Sample size is unrealistically large | MDE too small or baseline too low | Increase MDE (test bolder changes) or target a higher-traffic page |439| Test duration exceeds 6 weeks | Insufficient daily traffic | Consider qualitative methods, test bigger changes, or combine traffic from multiple pages |440| p-value hovers around 0.05 | Borderline significance | Do not stop early; run to planned sample size or extend 20% |441| Results significant but lift is tiny (<1%) | Overpowered test | Check practical significance alongside statistical significance |442| Segment results contradict overall | Simpson's paradox | Investigate segment composition; report both overall and segment results |443| Variant performs differently on mobile vs desktop | Device-specific UX issues | Design device-specific variants; increase per-segment sample size |444| Calculator produces negative CI | Very small samples or extreme rates | Ensure sufficient sample size; check data integrity |445446---447448## Success Criteria449450| Criterion | Target | How to Measure |451|-----------|--------|----------------|452| Tests reach planned sample size | 100% of tests | Compare actual vs planned sample at conclusion |453| False positive rate | <5% | Track post-implementation lift vs test prediction |454| Test velocity | 2+ tests per team per month | Count experiments documented per sprint |455| Documentation completeness | 100% of tests documented | Audit experiment records quarterly |456| Average test duration | <4 weeks | Measure start-to-conclusion calendar days |457| Decision quality | >80% of shipped variants hold gains at 90 days | Post-ship metric tracking |458459---460461## Scope & Limitations462463**In scope:**464- Hypothesis formulation and validation465- Sample size and power calculations466- Frequentist two-proportion z-tests467- A/B, A/B/n, and split URL test planning468- Segment-level analysis469- Pre/post test documentation470471**Out of scope:**472- Bayesian A/B testing methods (use dedicated Bayesian tools)473- Multi-armed bandit algorithms (require real-time allocation infrastructure)474- Multivariate testing (MVT) analysis (combinatorial explosion requires specialized tools)475- Server-side feature flag implementation (see engineering skills)476- Revenue-based metrics requiring transaction-level data477- Sequential testing / always-valid p-values (use Optimizely Stats Engine or similar)478479---480481## Integration Points482483| Tool / Platform | Integration Method | Use Case |484|-----------------|-------------------|----------|485| PostHog / Amplitude | JSON export from experiment_analyzer | Feed results into product analytics |486| Jira / Linear | experiment_planner JSON output | Create experiment tickets with metadata |487| Google Sheets | CSV export from experiment_analyzer | Share results with non-technical stakeholders |488| LaunchDarkly / Unleash | experiment_planner checklist | Pre-launch validation before feature flag rollout |489| Slack / Notion | Copy human-readable output | Async experiment status updates |490| CI/CD pipelines | `--json` flag on all scripts | Automated experiment health checks |
borghei