Experiment Design

Experiment Design

Types of Experiments

1. A/B Test (Two Variants)

What: Compare two versions (A vs B)

Example:

• Control (A): Blue "Buy Now" button
• Treatment (B): Green "Buy Now" button

When to Use:

• Testing single change
• Clear hypothesis
• Binary decision (ship or don't ship)

Pros:

• Simple to implement
• Easy to analyze
• Clear winner

Cons:

• Only tests one change
• Can't test interactions

2. Multivariate Test (Multiple Changes)

What: Test multiple changes simultaneously

Example:

• Variable 1: Button color (Blue, Green, Red)
• Variable 2: Button text ("Buy Now", "Add to Cart", "Get Started")
• Variants: 3 × 3 = 9 combinations

When to Use:

• Testing multiple elements
• Want to find best combination
• Have enough traffic

Pros:

• Test interactions between variables
• Find optimal combination

Cons:

• Requires much more traffic
• Complex analysis
• Longer test duration

3. Sequential Testing

What: Continuously monitor and stop early if clear winner

Example:

• Start A/B test
• Check results daily
• Stop when statistical significance reached (could be day 3 or day 14)

When to Use:

• Want to ship winners fast
• High traffic
• Using tools that support it (Statsig, GrowthBook)

Pros:

• Faster results
• Less opportunity cost

Cons:

• Requires special statistical methods
• Can't "peek" with traditional A/B tests

4. Holdout Groups (Long-Term Effects)

What: Keep small % of users on old experience permanently

Example:

• 95% of users: New feature
• 5% of users: Old experience (holdout)

When to Use:

• Measure long-term effects
• Detect delayed negative impacts
• Validate cumulative changes

Pros:

• Detects long-term issues
• Measures true impact

Cons:

• Some users get worse experience
• Requires ongoing monitoring

When to Experiment

✅ Experiment When:

1. Significant Features (High Impact)

   • Major redesign
   • New pricing model
   • Core flow changes

2. Uncertain Outcomes

   • Don't know if it will work
   • Conflicting opinions
   • No clear data

3. Multiple Solution Options

   • Two different approaches
   • Want to pick the best

4. Optimization Opportunities

   • Incremental improvements
   • Conversion optimization
   • Engagement optimization

❌ Don't Experiment When:

1. Obvious Bugs/Fixes

   • Broken functionality
   • Security issues
   • Legal compliance

2. Very Low Traffic

   • Can't reach statistical significance
   • Would take months

3. Trivial Changes

   • Copy typo fix
   • Minor styling adjustment

4. Ethical Issues

   • Manipulative dark patterns
   • Harmful to users

Experiment Design Process

Step 1: Define Hypothesis

Template:

"If we [change], then [metric] will [improve by X%], because [reasoning]."

Example:

"If we change the CTA button from blue to green, then click-through rate will increase by 10%, because green is more attention-grabbing."

Step 2: Choose Metrics

Primary Metric: What you're optimizing

• Example: Click-through rate

Secondary Metrics: Other important outcomes

• Example: Conversion rate, revenue per user

Counter Metrics: Watch for negatives

• Example: Bounce rate, time on page

Step 3: Determine Sample Size

Inputs:

• Baseline conversion rate: 5%
• Expected improvement: 10% relative lift (5% → 5.5%)
• Significance level: 0.05 (95% confidence)
• Power: 0.80 (80% chance of detecting effect)

Output:

• Sample size needed: ~31,000 users per variant

Tools:

• Evan Miller's calculator: https://www.evanmiller.org/ab-testing/sample-size.html
• Optimizely sample size calculator

Step 4: Set Test Duration

Factors:

• Sample size needed
• Daily traffic
• Weekly patterns (run at least 1-2 weeks)
• Business cycles

Example:

• Sample size: 31,000 per variant (62,000 total)
• Daily traffic: 5,000
• Duration: 62,000 / 5,000 = 12.4 days → Run for 2 weeks

Step 5: Design Variants

Control (A): Current experience Treatment (B): New experience

Best Practices:

• Change only one thing (for A/B test)
• Make change meaningful (not trivial)
• Ensure variants are distinct

Step 6: Launch Test

Checklist:

• Hypothesis documented
• Metrics instrumented
• Sample size calculated
• Randomization working
• QA tested both variants
• Monitoring dashboard ready

Step 7: Analyze Results

Check:

• Statistical significance (p < 0.05)
• Practical significance (is improvement meaningful?)
• Secondary metrics (any red flags?)
• Segment analysis (works for everyone?)

Step 8: Decide (Ship, Iterate, Kill)

Ship if:

• Positive, significant, no red flags

Iterate if:

• Mixed results, some segments good

Kill if:

• Negative, not significant, opportunity cost too high

Choosing Metrics

Primary Metric (What We're Optimizing)

Characteristics:

• Directly tied to hypothesis
• Sensitive to change
• Measurable in test duration

Examples:

• Click-through rate (CTR)
• Conversion rate
• Sign-up completion rate
• Time to first action

Bad Primary Metrics:

• Revenue (too noisy, delayed)
• Retention (takes too long to measure)
• NPS (survey-based, low sample)

Secondary Metrics (Guardrails, Side Effects)

Purpose: Ensure we're not breaking other things

Examples:

• Revenue per user
• Engagement (sessions per user)
• Feature adoption
• Customer satisfaction

Counter Metrics (Watch for Negatives)

Purpose: Detect unintended negative consequences

Examples:

• Bounce rate (users leaving immediately)
• Error rate (technical issues)
• Support tickets (confusion)
• Churn rate (users leaving)

Example: Checkout Flow Test

Hypothesis:

"If we reduce checkout from 5 steps to 3 steps, conversion will increase by 15%."

Metrics:

• Primary: Checkout conversion rate
• Secondary: Average order value, time to complete checkout
• Counter: Cart abandonment rate, error rate, support tickets

Statistical Significance

P-Value < 0.05 (95% Confidence)

What it Means:

• Less than 5% chance result is due to random chance
• 95% confident the effect is real

Example:

• Control: 5.0% conversion
• Treatment: 5.5% conversion
• P-value: 0.03 ✅ (< 0.05, statistically significant)

Interpretation:

"We're 95% confident that the treatment is better than control."

Statistical Power (80%+)

What it Means:

• 80% chance of detecting an effect if it exists
• Reduces false negatives

Example:

• Power: 80%
• Means: 20% chance of missing a real effect

Minimum Detectable Effect (MDE)

What it Means:

• Smallest effect size you can reliably detect
• Depends on sample size

Example:

• Baseline: 5% conversion
• Sample size: 10,000 per variant
• MDE: 0.5% absolute (10% relative)
• Can detect: 5.0% → 5.5% or larger

Trade-off:

• Larger sample size → Smaller MDE (detect smaller effects)
• Smaller sample size → Larger MDE (only detect big effects)

Sample Size Calculation

Formula (Simplified)

n = (Z_α/2 + Z_β)² × (p₁(1-p₁) + p₂(1-p₂)) / (p₁ - p₂)²

Where:
- n = sample size per variant
- Z_α/2 = 1.96 (for 95% confidence)
- Z_β = 0.84 (for 80% power)
- p₁ = baseline conversion rate
- p₂ = expected conversion rate

Example Calculation

Inputs:

• Baseline conversion rate (p₁): 5% = 0.05
• Expected improvement: 10% relative lift
• New conversion rate (p₂): 5.5% = 0.055
• Significance level (α): 0.05
• Power (1-β): 0.80

Calculation:

n = (1.96 + 0.84)² × (0.05×0.95 + 0.055×0.945) / (0.05 - 0.055)²
n = 7.84 × (0.0475 + 0.052) / 0.000025
n = 7.84 × 0.0995 / 0.000025
n ≈ 31,200 per variant

Total sample size: 62,400 users

Using Online Calculators

Evan Miller's Calculator:

1. Go to https://www.evanmiller.org/ab-testing/sample-size.html
2. Enter baseline conversion rate: 5%
3. Enter minimum detectable effect: 10% (relative)
4. Get sample size: ~31,000 per variant

Optimizely Calculator:

1. Go to Optimizely sample size calculator
2. Enter baseline: 5%
3. Enter minimum detectable effect: 0.5% (absolute)
4. Get sample size: ~31,000 per variant

Test Duration

Minimum Duration: 1-2 Weeks

Why:

• Capture weekly patterns (weekday vs weekend)
• Avoid day-of-week bias
• Account for user behavior cycles

Example:

• Don't run Monday-Wednesday only
• Run at least Monday-Sunday (1 full week)

Full Business Cycles

Examples:

• E-commerce: Include payday (1st and 15th of month)
• B2B SaaS: Include full week (avoid Friday-only)
• Seasonal: Avoid holidays (unless testing holiday-specific)

Enough Data for Significance

Formula:

Duration = Sample Size Needed / Daily Traffic

Example:

• Sample size: 62,000 total
• Daily traffic: 5,000
• Duration: 62,000 / 5,000 = 12.4 days
• Run for: 2 weeks (14 days)

Not Too Long (Opportunity Cost)

Trade-off:

• Longer test = More confidence
• Longer test = Delayed learnings, slower iteration

Guideline:

• Most tests: 1-4 weeks
• High-traffic sites: 1-2 weeks
• Low-traffic sites: 2-4 weeks
• Don't run > 1 month (diminishing returns)

Experiment Variants

Control (Current Experience)

What: The existing experience

Example:

• Current checkout flow (5 steps)
• Current button color (blue)
• Current pricing page

Purpose: Baseline for comparison

Treatment (New Experience)

What: The proposed change

Example:

• New checkout flow (3 steps)
• New button color (green)
• New pricing page

Purpose: Test hypothesis

Multiple Treatments (If Testing Different Approaches)

Example:

• Control: 5-step checkout
• Treatment A: 3-step checkout (combine steps)
• Treatment B: 1-page checkout (all on one page)

Traffic Split:

• Control: 33%
• Treatment A: 33%
• Treatment B: 34%

Analysis:

• Compare each treatment to control
• Compare treatments to each other

Randomization

User-Level Randomization (Consistent Experience)

What: Each user always sees same variant

How:

const variant = hashUserId(userId) % 2 === 0 ? 'control' : 'treatment';

When to Use:

• Logged-in users
• Want consistent experience
• Testing flows (multi-step)

Pros:

• Consistent experience
• No confusion

Cons:

• Requires user ID

Session-Level (For Anonymous Users)

What: Each session sees same variant (but different sessions can differ)

How:

const variant = hashSessionId(sessionId) % 2 === 0 ? 'control' : 'treatment';

When to Use:

• Anonymous users
• Single-page tests

Pros:

• Works for anonymous users

Cons:

• Same user can see different variants across sessions

Stratified Sampling (For Segments)

What: Ensure even distribution across segments

Example:

• Segment 1: Free users (50% control, 50% treatment)
• Segment 2: Paid users (50% control, 50% treatment)

Why:

• Avoid imbalanced segments
• Enable segment analysis

Common Pitfalls

1. Peeking (Stopping Test Early When "Winning")

Problem:

Day 3: Treatment is winning! (p = 0.04) → Ship it!
Day 7: Treatment is losing... (p = 0.12) → Oops.

Why It's Bad:

• Increases false positive rate
• P-value fluctuates during test

Solution:

• Decide sample size upfront
• Don't look until test completes
• Or use sequential testing (proper method)

2. Sample Ratio Mismatch (Uneven Splits)

Problem:

Expected: 50% control, 50% treatment
Actual: 48% control, 52% treatment

Why It's Bad:

• Indicates randomization bug
• Results may be invalid

Solution:

• Check sample ratio before analyzing
• Investigate if mismatch > 1%

3. Novelty Effect (Users Trying New Thing)

Problem:

Week 1: Treatment is winning! (+20%)
Week 4: Treatment is same as control (0%)

Why It's Bad:

• Users try new thing out of curiosity
• Effect fades over time

Solution:

• Run test longer (2-4 weeks)
• Use holdout group for long-term measurement
• Segment by new vs returning users

4. Seasonality (Testing During Holidays)

Problem:

Test during Black Friday: +50% conversion
Test during normal week: +5% conversion

Why It's Bad:

• Holiday behavior is different
• Results don't generalize

Solution:

• Avoid testing during holidays
• Or run test across multiple weeks (include holiday + normal)

Sequential Testing

What is Sequential Testing?

Traditional A/B Test:

• Decide sample size upfront
• Run until sample size reached
• Analyze once at end

Sequential Testing:

• Monitor continuously
• Stop early if clear winner
• Adjust significance threshold

How It Works

Algorithm:

• Use adjusted significance threshold (not 0.05)
• Account for multiple looks
• Stop when threshold crossed

Example (Simplified):

Day 1: p = 0.10 → Continue
Day 3: p = 0.03 → Continue
Day 5: p = 0.001 → Stop! (clear winner)

Tools That Support Sequential Testing

• Statsig: Built-in sequential testing
• GrowthBook: Bayesian statistics
• Optimizely: Stats Engine (sequential)

Benefits

• Faster results (stop early if clear winner)
• Less opportunity cost
• Detect large effects quickly

Drawbacks

• Requires special tools
• Can't use traditional p-value
• More complex

Holdout Groups

What is a Holdout Group?

Definition: Small % of users kept on old experience permanently

Example:

• 95% of users: New feature
• 5% of users: Old experience (holdout)

Why Use Holdout Groups?

Measure Long-Term Effects:

• A/B test shows +10% conversion in 2 weeks
• Holdout shows +5% conversion after 6 months
• Learning: Effect diminishes over time

Detect Delayed Negative Impacts:

• A/B test shows +15% signups
• Holdout shows +10% churn after 3 months
• Learning: Feature attracts wrong users

How Long to Keep Holdout?

Guideline:

• 1-3 months for most features
• 6-12 months for major changes
• Permanent for critical features

When to Remove Holdout?

Remove if:

• No long-term differences detected
• Opportunity cost too high (5% of users on worse experience)
• Feature is critical (everyone should have it)

Experiment Analysis

Step 1: Compare Primary Metric

Example:

• Control: 5.0% conversion
• Treatment: 5.5% conversion
• Lift: +10% relative
• P-value: 0.03 ✅

Decision: Treatment is statistically significantly better.

Step 2: Check Secondary Metrics

Example:

• Revenue per user: $10.50 (control) vs $11.20 (treatment) ✅
• Time to checkout: 3.2 min (control) vs 2.8 min (treatment) ✅

Decision: Secondary metrics also improved.

Step 3: Check Counter Metrics

Example:

• Bounce rate: 30% (control) vs 32% (treatment) ⚠️
• Error rate: 0.5% (control) vs 0.5% (treatment) ✅

Decision: Slight increase in bounce rate, investigate.

Step 4: Segment Analysis

Did it work for everyone?

Learning: Works great for mobile and free users, minimal impact on paid users.

Step 5: Statistical Significance

Check:

• P-value < 0.05 ✅
• Confidence interval doesn't include 0 ✅

Example:

• Lift: +10%
• 95% CI: [+5%, +15%]
• Interpretation: We're 95% confident the true lift is between 5% and 15%.

Step 6: Practical Significance

Is the improvement meaningful?

Example:

• Statistically significant: Yes (p = 0.04)
• Lift: +0.1% (5.0% → 5.005%)
• Decision: Not practically significant (too small to matter)

Guideline:

• Small lift but high volume → Ship (e.g., +0.1% on 1M users = 1,000 more conversions)
• Large lift but low volume → Maybe ship (e.g., +50% on 100 users = 50 more conversions)

Decision Framework

Ship If:

✅ Positive: Treatment is better than control ✅ Significant: P-value < 0.05 ✅ No Red Flags: Secondary and counter metrics look good ✅ Works for Key Segments: At least works for majority

Example:

• Conversion: +10% (p = 0.03) ✅
• Revenue: +8% (p = 0.05) ✅
• Bounce rate: No change ✅
• Works for mobile and desktop ✅
• Decision: Ship!

Iterate If:

⚠️ Mixed Results: Some metrics up, some down ⚠️ Works for Some Segments Only: E.g., only mobile, not desktop ⚠️ Close to Significance: P = 0.06 (just missed)

Example:

• Conversion: +10% (p = 0.03) ✅
• Revenue: -5% (p = 0.08) ⚠️
• Decision: Iterate. Conversion is up but revenue is down. Investigate why.

Kill If:

❌ Negative: Treatment is worse than control ❌ Not Significant: P-value > 0.05 ❌ Opportunity Cost Too High: Could be working on better ideas

Example:

• Conversion: +2% (p = 0.15) ❌
• Took 4 weeks to test
• Decision: Kill. Not significant, move on to next idea.

Tools

Feature Flags

LaunchDarkly:

• Feature flag management
• Gradual rollouts
• Kill switches

Split.io:

• Feature flags + experimentation
• Real-time metrics

Unleash:

• Open-source feature flags
• Self-hosted option

Experimentation Platforms

Optimizely:

• Full-stack experimentation
• Visual editor for web
• Stats Engine (sequential testing)

VWO (Visual Website Optimizer):

• A/B testing for web
• Heatmaps, session recordings
• Visual editor

GrowthBook:

• Open-source experimentation
• Bayesian statistics
• Feature flags

Statsig:

• Modern experimentation platform
• Sequential testing
• Free tier

Analytics

Amplitude:

• Product analytics
• Funnel analysis
• Cohort analysis

Mixpanel:

• Event-based analytics
• A/B test analysis
• Retention analysis

PostHog:

• Open-source product analytics
• Feature flags
• Session replay

A/B Testing for Engineers

1. Feature Flag Implementation

Node.js (LaunchDarkly):

const LaunchDarkly = require('launchdarkly-node-server-sdk');

const client = LaunchDarkly.init(process.env.LAUNCHDARKLY_SDK_KEY);

await client.waitForInitialization();

app.get('/checkout', async (req, res) => {
  const user = {
    key: req.user.id,
    email: req.user.email,
    custom: {
      plan: req.user.plan
    }
  };
  
  const showNewCheckout = await client.variation('new-checkout-flow', user, false);
  
  if (showNewCheckout) {
    res.render('checkout-new');
  } else {
    res.render('checkout-old');
  }
});

Python (Statsig):

from statsig import statsig

statsig.initialize(os.environ['STATSIG_SERVER_KEY'])

@app.route('/checkout')
def checkout():
    user = {
        'userID': current_user.id,
        'email': current_user.email,
        'custom': {
            'plan': current_user.plan
        }
    }
    
    show_new_checkout = statsig.check_gate(user, 'new_checkout_flow')
    
    if show_new_checkout:
        return render_template('checkout_new.html')
    else:
        return render_template('checkout_old.html')

2. Metric Instrumentation

Segment (Event Tracking):

const Analytics = require('analytics-node');
const analytics = new Analytics(process.env.SEGMENT_WRITE_KEY);

// Track checkout started
analytics.track({
  userId: user.id,
  event: 'Checkout Started',
  properties: {
    variant: showNewCheckout ? 'treatment' : 'control',
    cart_value: cart.total,
    items_count: cart.items.length
  }
});

// Track checkout completed
analytics.track({
  userId: user.id,
  event: 'Checkout Completed',
  properties: {
    variant: showNewCheckout ? 'treatment' : 'control',
    order_id: order.id,
    revenue: order.total
  }
});

3. Data Pipeline

Architecture:

Application
    ↓ (events)
Segment
    ↓ (forwards to)
├── Amplitude (analytics)
├── Mixpanel (analytics)
├── Data Warehouse (BigQuery, Snowflake)
└── Statsig (experimentation)

4. Results Dashboard

Grafana Dashboard:

{
  "dashboard": {
    "title": "A/B Test: New Checkout Flow",
    "panels": [
      {
        "title": "Conversion Rate by Variant",
        "targets": [
          {
            "expr": "sum(checkout_completed{variant='control'}) / sum(checkout_started{variant='control'})",
            "legendFormat": "Control"
          },
          {
            "expr": "sum(checkout_completed{variant='treatment'}) / sum(checkout_started{variant='treatment'})",
            "legendFormat": "Treatment"
          }
        ]
      },
      {
        "title": "Sample Size",
        "targets": [
          {
            "expr": "sum(checkout_started{variant='control'})",
            "legendFormat": "Control"
          },
          {
            "expr": "sum(checkout_started{variant='treatment'})",
            "legendFormat": "Treatment"
          }
        ]
      }
    ]
  }
}

Real Experiment Examples

Example 1: Button Color Test (Classic)

Hypothesis:

"If we change the CTA button from blue to orange, click-through rate will increase by 10%, because orange is more attention-grabbing."

Test:

• Control: Blue button
• Treatment: Orange button
• Sample size: 10,000 per variant
• Duration: 1 week

Results:

• Control: 5.2% CTR
• Treatment: 5.7% CTR
• Lift: +9.6%
• P-value: 0.04 ✅

Decision: Ship orange button.

Example 2: Checkout Flow Optimization

Hypothesis:

"If we reduce checkout from 5 steps to 3 steps, conversion will increase by 15%, because users abandon due to flow length."

Test:

• Control: 5-step checkout
• Treatment: 3-step checkout (combined steps)
• Sample size: 50,000 per variant
• Duration: 2 weeks

Results:

• Control: 8.5% conversion
• Treatment: 9.8% conversion
• Lift: +15.3%
• P-value: 0.001 ✅

Secondary Metrics:

• Time to checkout: 4.2 min → 3.1 min ✅
• Error rate: 2.1% → 1.8% ✅

Decision: Ship 3-step checkout.

Example 3: Pricing Page Variants

Hypothesis:

"If we show annual pricing first (instead of monthly), annual plan adoption will increase by 25%, because anchoring effect."

Test:

• Control: Monthly pricing shown first
• Treatment: Annual pricing shown first
• Sample size: 20,000 per variant
• Duration: 3 weeks

Results:

• Control: 12% annual adoption
• Treatment: 18% annual adoption
• Lift: +50%
• P-value: 0.001 ✅

Counter Metrics:

• Overall conversion: 10.5% → 10.2% ⚠️ (slight drop)

Decision: Ship, but monitor overall conversion.

Example 4: Onboarding Flow

Hypothesis:

"If we add an interactive tutorial in onboarding, activation rate will increase by 30%, because users don't know how to get started."

Test:

• Control: No tutorial
• Treatment: Interactive tutorial (5 steps)
• Sample size: 15,000 per variant
• Duration: 2 weeks

Results:

• Control: 25% activation rate
• Treatment: 28% activation rate
• Lift: +12%
• P-value: 0.08 ❌ (not significant)

Segment Analysis:

• New users: +20% (p = 0.03) ✅
• Returning users: +2% (p = 0.5) ❌

Decision: Iterate. Show tutorial only to new users.

Advanced: Bayesian A/B Testing

Traditional (Frequentist) A/B Testing

Approach:

• Null hypothesis: No difference between A and B
• P-value: Probability of seeing this result if null is true
• Reject null if p < 0.05

Interpretation:

"There's a 95% chance the result is not due to random chance."

Bayesian A/B Testing

Approach:

• Prior belief: What we believe before test
• Likelihood: Data from test
• Posterior belief: Updated belief after test

Interpretation:

"There's a 95% probability that B is better than A."

Benefits of Bayesian

1. Easier to Interpret:

   • "95% probability B is better" (intuitive)
   • vs "p = 0.03" (confusing)

2. Can Stop Early:

   • No peeking problem
   • Stop when confident enough

3. Incorporates Prior Knowledge:

   • Use historical data
   • More accurate with small samples

Tools That Use Bayesian

• GrowthBook: Bayesian by default
• VWO: Bayesian engine option
• Google Optimize: Bayesian (deprecated)

Example

Test:

• Control: 5.0% conversion (1000 users)
• Treatment: 5.5% conversion (1000 users)

Frequentist:

• P-value: 0.15 (not significant)
• Decision: Can't conclude

Bayesian:

• Probability B > A: 87%
• Expected lift: +10%
• Decision: Likely better, but not confident enough (need 95%)

Summary

Quick Reference

Experiment Types:

• A/B test: Two variants
• Multivariate: Multiple changes
• Sequential: Stop early
• Holdout: Long-term measurement

When to Experiment:

• Significant features
• Uncertain outcomes
• Multiple options
• Optimization

Process:

1. Define hypothesis
2. Choose metrics
3. Calculate sample size
4. Set duration
5. Design variants
6. Launch
7. Analyze
8. Decide

Metrics:

• Primary: What we're optimizing
• Secondary: Guardrails
• Counter: Watch for negatives

Statistical Significance:

• P-value < 0.05
• Power > 80%
• Minimum detectable effect

Common Pitfalls:

• Peeking
• Sample ratio mismatch
• Novelty effect
• Seasonality

Decision Framework:

• Ship: Positive, significant, no red flags
• Iterate: Mixed results
• Kill: Negative, not significant

Tools:

• Feature flags: LaunchDarkly, Split.io
• Experimentation: Optimizely, Statsig, GrowthBook
• Analytics: Amplitude, Mixpanel, PostHog