Ab Test Setup

A/B Test Setup - Experimentation Design & Analysis

Category: Product Team Tags: A/B testing, experiments, statistical significance, sample size, feature flags, hypothesis testing

Overview

A/B Test Setup provides the complete framework for designing experiments that produce statistically valid, actionable results. Most A/B tests fail not because the variant was wrong, but because the test was poorly designed: wrong sample size, wrong metric, or someone peeked at results and stopped early. This skill prevents those mistakes.

Clarify First

Before designing the experiment, confirm these inputs. If any is unknown or vague, ASK — do not assume:

• Primary metric + minimum detectable effect — the one success metric and smallest lift worth detecting (drives sample size, duration, and metric selection)
• Baseline conversion rate — current rate for the primary metric (sets required sample size per variant)
• Available traffic to the test surface — daily eligible visitors (decides whether the test is feasible or needs a bolder change / qualitative method)
• The change and its rationale — what varies and the data behind it (drives the hypothesis and variant design)

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.

The Experiment Lifecycle

1. HYPOTHESIZE  →  2. DESIGN  →  3. CALCULATE  →  4. IMPLEMENT
       ↑                                                    │
       │                                                    ▼
7. ITERATE  ←  6. DOCUMENT  ←  5. ANALYZE  ←  [Run to completion]

Step 1: Hypothesis Formulation

The Hypothesis Template

Because [observation or data point],
we believe [specific change]
will cause [measurable outcome]
for [defined audience segment].

We'll know this is true when [primary metric] changes by [minimum detectable effect].
We'll watch [guardrail metrics] to ensure no negative impact.

Good vs Bad Hypotheses

Hypothesis Sources (Where to Find Test Ideas)

Step 2: Test Design

Test Types

Default recommendation: Standard A/B test. Only use A/B/n or MVT when you have enough traffic and a specific need.

What to Test (By Impact)

Metric Selection

Every test needs three types of metrics:

Primary Metric (1 only)

• The single metric that determines success
• Directly tied to the hypothesis
• Must be measurable within the test duration
• Examples: signup rate, click-through rate, purchase rate

Secondary Metrics (2-3)

• Explain why the primary metric moved
• Provide context for decision-making
• Examples: time on page, scroll depth, feature adoption rate

Guardrail Metrics (1-3)

• Things that must NOT get worse
• Stop the test if significantly negative
• Examples: error rate, support ticket volume, page load time, refund rate

Step 3: Sample Size Calculation

Quick Reference Table

Minimum visitors PER VARIANT needed (95% confidence, 80% power):

Duration Calculation

Duration (days) = (Sample size per variant * Number of variants) / Daily traffic to test page

Minimum duration: 7 days (to capture day-of-week effects) Maximum recommended: 6 weeks (beyond this, external factors contaminate results)

What If You Don't Have Enough Traffic?

Step 4: Implementation

Client-Side Implementation

JavaScript modifies the page after initial render.

Pros: Quick to implement, no deploy needed Cons: Can cause flicker (flash of original content), blocked by ad blockers Tools: PostHog, Optimizely, VWO, Google Optimize

Anti-flicker pattern:

// Add to <head> before any rendering
<style>.ab-test-hide { opacity: 0 !important; }</style>
<script>document.documentElement.classList.add('ab-test-hide');</script>

// In your test script (runs after variant assignment):
document.documentElement.classList.remove('ab-test-hide');

Server-Side Implementation

Variant determined before page renders. No flicker, no client-side dependency.

Pros: No flicker, not blocked by ad blockers, works for logged-in features Cons: Requires engineering work, deploy needed Tools: PostHog, LaunchDarkly, Split, Unleash, custom feature flags

Basic feature flag pattern:

# Server-side variant assignment
def get_variant(user_id: str, experiment: str) -> str:
    # Deterministic hash ensures same user always sees same variant
    hash_input = f"{user_id}:{experiment}"
    hash_value = hashlib.md5(hash_input.encode()).hexdigest()
    bucket = int(hash_value[:8], 16) % 100

    if bucket < 50:
        return "control"
    else:
        return "variant"

Traffic Allocation

Critical rules:

• Users must see the same variant on every visit (sticky assignment by user ID or cookie)
• Allocation must be balanced across time of day and day of week
• Never change allocation mid-test

Step 5: Running the Test

Pre-Launch Checklist

• Hypothesis documented with primary metric and minimum detectable effect
• Sample size calculated, expected duration estimated
• Both variants implemented and QA'd on all device types
• Tracking verified (events fire correctly for both variants)
• No other tests running on the same page/feature
• Stakeholders informed of test duration and "no peeking" rule
• External factor calendar checked (no major launches, holidays, press)

During the Test

DO:

• Monitor for technical errors (variant not rendering, tracking broken)
• Check that traffic split is balanced daily
• Document any external events that might affect results

DO NOT:

• Look at results before reaching sample size ("peeking problem")
• Make changes to either variant
• Add traffic from new sources mid-test
• Stop the test early because one variant "looks like it's winning"

The Peeking Problem (Critical)

Looking at results before reaching the planned sample size and stopping because one variant looks better leads to a 25-40% false positive rate (vs the intended 5%).

Why: Statistical significance fluctuates wildly with small samples. A variant can show p < 0.05 at 20% of planned sample size and p > 0.30 at full sample.

Solutions:

1. Pre-commit to sample size and do not check results until reached
2. If you must monitor: use sequential testing methods (group sequential design, always-valid p-values)
3. Set calendar reminder for expected completion date -- that is when you look

Step 6: Analysis

Analysis Checklist

1. Did we reach planned sample size? If not, results are preliminary only.
2. Is it statistically significant? p < 0.05 = 95% confidence the difference is real.
3. What's the confidence interval? Tells you the range of likely true effect.
4. Is the effect size meaningful? A 0.1% lift that's "significant" may not be worth implementing.
5. Are secondary metrics consistent? Do they support the primary result?
6. Any guardrail violations? Did anything get worse?
7. Segment analysis: Different results for mobile vs desktop? New vs returning?

Interpreting Results

Common Analysis Mistakes

Step 7: Documentation

Every test must be documented, regardless of outcome.

Test Documentation Template

EXPERIMENT: [Name]
DATE: [Start] to [End]
OWNER: [Name]

HYPOTHESIS:
Because [observation], we believed [change] would cause [outcome] for [audience].

VARIANTS:
- Control: [description]
- Variant: [description + screenshot]

METRICS:
- Primary: [metric] (baseline: [X]%, MDE: [Y]%)
- Secondary: [metrics]
- Guardrails: [metrics]

RESULTS:
- Sample size: [actual] / [planned]
- Duration: [X] days
- Primary metric: Control [X]% vs Variant [Y]% (p = [Z], CI: [range])
- Secondary metrics: [results]
- Guardrails: [all clear / violation noted]

DECISION: [Ship variant / Keep control / Iterate]

LEARNINGS:
- [What we learned about our users]
- [What we'd do differently next time]

Experiment Prioritization Framework

ICE Scoring

ICE Score = (Impact + Confidence + Ease) / 3

Rank all test ideas by ICE score. Run highest first.

Test Backlog Template

Proactive Triggers

• Someone debates between two design options: propose an A/B test instead of opinionating
• Conversion rate mentioned as underperforming: offer to design a test, not guess at solutions
• Pricing page changes discussed: always test pricing changes with guardrail metrics
• Post-launch of any feature: propose follow-up experiment to optimize
• "Let's just try it and see": redirect to structured hypothesis before implementation

Related Skills

Tool Reference

sample_size_calculator.py

Calculates required sample size per variant using the normal approximation to the two-proportion z-test. Includes Bonferroni correction for multi-variant tests and duration estimation.

python scripts/sample_size_calculator.py --baseline 0.05 --mde 0.10
python scripts/sample_size_calculator.py --baseline 0.12 --mde 0.15 --power 0.9 --daily-traffic 5000
python scripts/sample_size_calculator.py --baseline 0.05 --mde 0.10 --variants 3 --json

experiment_analyzer.py

Analyzes A/B test results using the two-proportion z-test with confidence intervals and segment breakdown.

CSV format: variant,visitors,conversions,segment

python scripts/experiment_analyzer.py sample
python scripts/experiment_analyzer.py results.csv
python scripts/experiment_analyzer.py results.csv --alpha 0.01 --json

experiment_planner.py

Generates a structured experiment plan from a hypothesis text, including metric selection, sample size, timeline, risks, and documentation template.

python scripts/experiment_planner.py --hypothesis "Larger CTA will increase signups by 15%"
python scripts/experiment_planner.py -H "Simplified checkout boosts conversions" -b 0.08 -m 0.15 -d 3000
python scripts/experiment_planner.py -H "New pricing page" --json

Troubleshooting

Success Criteria

Scope & Limitations

In scope:

• Hypothesis formulation and validation
• Sample size and power calculations
• Frequentist two-proportion z-tests
• A/B, A/B/n, and split URL test planning
• Segment-level analysis
• Pre/post test documentation

Out of scope:

• Bayesian A/B testing methods (use dedicated Bayesian tools)
• Multi-armed bandit algorithms (require real-time allocation infrastructure)
• Multivariate testing (MVT) analysis (combinatorial explosion requires specialized tools)
• Server-side feature flag implementation (see engineering skills)
• Revenue-based metrics requiring transaction-level data
• Sequential testing / always-valid p-values (use Optimizely Stats Engine or similar)

Integration Points