Ab Testing

A/B Testing

You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.

Initial Assessment

Check for product marketing context first: If .agents/product-marketing-context.md exists, read it before asking questions.

Before designing a test, understand:

1. Test Context — What are you trying to improve? What change are you considering?
2. Current State — Baseline conversion rate? Current traffic volume?
3. Constraints — Technical complexity? Timeline? Tools available?

Core Principles

1. Start with a Hypothesis — Not just "let's see what happens." Specific prediction based on reasoning or data.
2. Test One Thing — Single variable per test. Otherwise you don't know what worked.
3. Statistical Rigor — Pre-determine sample size. Don't peek and stop early.
4. Measure What Matters — Primary metric tied to business value. Secondary + guardrail metrics.

Hypothesis Framework

Structure

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

Example

Weak: "Changing the button color might increase clicks."

Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."

Template alternativo (IF/THEN)

IF we [change],
THEN [metric] will [increase/decrease] by [amount],
BECAUSE [rationale based on user behavior/data].

When to A/B Test

Test Hierarchy (Impact vs Effort)

HIGH IMPACT / LOW EFFORT (Test First)
├── Headlines
├── CTA text
├── CTA color/size
├── Hero image
└── Form fields (remove/add)

MEDIUM IMPACT / MEDIUM EFFORT
├── Page layout
├── Social proof placement
├── Pricing presentation
├── Above-fold content
└── Form steps

LOW IMPACT / HIGH EFFORT (Test Last)
├── Full redesign
├── Navigation changes
└── Brand changes

Test Types

Sample Size

Quick Reference (95% confidence, 80% power)

Sample size per variant

Formula

n = (2 * (Zα + Zβ)² * p * (1-p)) / MDE²

Where:
- n = sample size per variant
- Zα = Z-score for significance (1.96 for 95%)
- Zβ = Z-score for power (0.84 for 80%)
- p = baseline conversion rate
- MDE = minimum detectable effect

Duration Estimator

Duration (days) = (Sample Size × 2) / Daily Visitors

Example:
- Need 3,000 per variant = 6,000 total
- 500 visitors/day
- Duration = 6,000 / 500 = 12 days

Calculators:

• Evan Miller's
• Optimizely's

Metrics Selection

Primary Metric

• Single metric that matters most
• Directly tied to hypothesis
• What you'll use to call the test

Secondary Metrics

• Support primary metric interpretation
• Explain why/how the change worked

Guardrail Metrics

• Things that shouldn't get worse
• Stop test if significantly negative

Example: Pricing Page Test

• Primary: Plan selection rate
• Secondary: Time on page, plan distribution
• Guardrail: Support tickets, refund rate

Test Documentation

## Test: [Test Name]
**ID**: TEST-2026-001
**Status**: [Planning | Running | Analyzing | Complete]

### Hypothesis
[Hypothesis statement]

### Variants
| Variant | Description |
|---------|-------------|
| Control (A) | [Current state] |
| Variant (B) | [Change description] |

### Metrics
- **Primary**: [e.g., CTA click rate]
- **Secondary**: [e.g., Form submission rate, Bounce rate]
- **Guardrail**: [e.g., Page load time, Error rate]

### Targeting
- **Audience**: [All visitors | Segment]
- **Traffic Split**: 50/50
- **Device**: [All | Desktop | Mobile]

### Sample Size
- **Required**: [Calculate based on MDE]
- **Expected Duration**: [X days]

### Results
| Metric | Control | Variant | Lift | Confidence |
|--------|---------|---------|------|------------|
| Primary | X% | Y% | +Z% | 95% |

### Decision
[Ship Variant | Keep Control | Iterate | Inconclusive]

### Learnings
[What we learned for future tests]

Implementation (Next.js)

Feature Flag Approach

// lib/ab-test.ts
import { cookies } from 'next/headers'

export type ABVariant = 'control' | 'variant'

interface ABTest {
  id: string
  name: string
  variants: ABVariant[]
  trafficAllocation: number // 0-1
}

const activeTests: ABTest[] = [
  {
    id: 'hero-headline-2026-01',
    name: 'Hero Headline Test',
    variants: ['control', 'variant'],
    trafficAllocation: 0.5,
  },
]

export function getABVariant(testId: string): ABVariant {
  const cookieStore = cookies()
  const cookieName = `ab_${testId}`

  const existing = cookieStore.get(cookieName)
  if (existing) return existing.value as ABVariant

  const test = activeTests.find(t => t.id === testId)
  if (!test) return 'control'

  const variant: ABVariant = Math.random() < test.trafficAllocation ? 'variant' : 'control'
  return variant
}

export function setABCookie(testId: string, variant: ABVariant) {
  cookies().set(`ab_${testId}`, variant, {
    httpOnly: true,
    secure: process.env.NODE_ENV === 'production',
    sameSite: 'lax',
    maxAge: 60 * 60 * 24 * 30,
  })
}

Server Component Usage

// app/page.tsx
import { getABVariant, setABCookie } from '@/lib/ab-test'
import { trackEvent } from '@/lib/analytics'

export default function HomePage() {
  const testId = 'hero-headline-2026-01'
  const variant = getABVariant(testId)
  setABCookie(testId, variant)

  trackEvent('ab_test_exposure', { test_id: testId, variant })

  return <Hero variant={variant} />
}

Hero Component with Variants

// components/hero.tsx
interface HeroProps {
  variant: 'control' | 'variant'
}

const headlines = {
  control: {
    h1: 'Build Better Frontend Applications',
    subhead: 'Professional development services for modern web apps.',
  },
  variant: {
    h1: 'Ship 3x Faster with Expert Frontend',
    subhead: 'Join 500+ teams who reduced time-to-market by 60%.',
  },
}

export function Hero({ variant }: HeroProps) {
  const copy = headlines[variant]
  return (
    <section className="hero">
      <h1>{copy.h1}</h1>
      <p>{copy.subhead}</p>
      <TrackedCTA id="hero-cta" location="hero" variant={variant}>
        Start Free Trial
      </TrackedCTA>
    </section>
  )
}

Client-Side Hook (for SPA)

// hooks/use-ab-test.ts
'use client'

import { useEffect, useState } from 'react'
import { analytics } from '@/lib/analytics'

export function useABTest(testId: string): 'control' | 'variant' | null {
  const [variant, setVariant] = useState<'control' | 'variant' | null>(null)

  useEffect(() => {
    const cookieName = `ab_${testId}`
    const existing = document.cookie
      .split('; ')
      .find(row => row.startsWith(cookieName))
      ?.split('=')[1]

    if (existing) {
      setVariant(existing as 'control' | 'variant')
      return
    }

    const newVariant = Math.random() < 0.5 ? 'variant' : 'control'
    document.cookie = `${cookieName}=${newVariant}; max-age=${60 * 60 * 24 * 30}; path=/`
    setVariant(newVariant)

    analytics.trackEvent?.('ab_test_exposure', { test_id: testId, variant: newVariant })
  }, [testId])

  return variant
}

Running the Test

Pre-Launch Checklist

• Hypothesis documented
• Primary metric defined
• Sample size calculated
• Variants implemented correctly
• Tracking verified
• QA completed on all variants

During the Test

DO:

• Monitor for technical issues
• Check segment quality
• Document external factors

DON'T:

• Peek at results and stop early (the peeking problem — leads to false positives)
• Make changes to variants mid-test
• Add traffic from new sources

Statistical Analysis

Significance Calculator

// lib/ab-stats.ts

interface ABResult {
  control: { visitors: number; conversions: number }
  variant: { visitors: number; conversions: number }
}

export function calculateSignificance(result: ABResult) {
  const { control, variant } = result

  const p1 = control.conversions / control.visitors
  const p2 = variant.conversions / variant.visitors

  const pooledP = (control.conversions + variant.conversions) /
                  (control.visitors + variant.visitors)

  const se = Math.sqrt(
    pooledP * (1 - pooledP) * (1 / control.visitors + 1 / variant.visitors)
  )

  const zScore = (p2 - p1) / se
  const pValue = 2 * (1 - normalCDF(Math.abs(zScore)))
  const lift = ((p2 - p1) / p1) * 100
  const confidence = (1 - pValue) * 100

  return {
    controlRate: p1 * 100,
    variantRate: p2 * 100,
    lift,
    pValue,
    confidence,
    isSignificant: pValue < 0.05,
    recommendation: getRecommendation(lift, pValue),
  }
}

function normalCDF(x: number): number {
  const a1 = 0.254829592, a2 = -0.284496736, a3 = 1.421413741
  const a4 = -1.453152027, a5 = 1.061405429, p = 0.3275911

  const sign = x < 0 ? -1 : 1
  x = Math.abs(x) / Math.sqrt(2)
  const t = 1.0 / (1.0 + p * x)
  const y = 1.0 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * Math.exp(-x * x)
  return 0.5 * (1.0 + sign * y)
}

function getRecommendation(lift: number, pValue: number): string {
  if (pValue >= 0.05) return 'INCONCLUSIVE - Need more data'
  if (lift > 0) return 'SHIP VARIANT - Statistically significant improvement'
  if (lift < 0) return 'KEEP CONTROL - Variant performed worse'
  return 'NO DIFFERENCE - Consider other factors'
}

Interpreting Results

Anti-Patterns

Statistical Mistakes

Implementation Mistakes

Test Velocity Framework

Prioritization (ICE Score)

ICE = (Impact + Confidence + Ease) / 3

Impact: 1-10 (potential uplift)
Confidence: 1-10 (likelihood of success)
Ease: 1-10 (implementation effort, inverted)

Test Roadmap Template

## Q1 2026 Test Roadmap

### Month 1
| Test | ICE | Status |
|------|-----|--------|
| Hero headline | 8.3 | Running |
| CTA button color | 6.0 | Planned |

### Month 2
| Test | ICE | Status |
|------|-----|--------|
| Form length | 7.5 | Planned |
| Social proof format | 7.0 | Planned |

Reporting Template

## A/B Test Report: [Test Name]

**Test ID**: TEST-2026-001
**Duration**: Jan 1 - Jan 14, 2026
**Traffic**: 12,450 visitors (6,225 per variant)

### Results Summary

| Metric | Control | Variant | Lift | Confidence |
|--------|---------|---------|------|------------|
| **Primary: CTA CTR** | 4.2% | 5.1% | **+21.4%** | 97.3% ✅ |
| Secondary: Form Starts | 2.1% | 2.4% | +14.3% | 89.2% |
| Guardrail: Bounce Rate | 42% | 41% | -2.4% | 65% |

### Recommendation
**SHIP VARIANT** - The new headline showed a statistically significant 21% improvement.

### Key Learnings
1. Benefit-focused headlines outperform feature-focused
2. Specific metrics ("3x faster") increase credibility

### Next Steps
- [ ] Roll out variant to 100% traffic
- [ ] Plan follow-up test on CTA button text

Tools & Platforms

Free/Open Source

• PostHog (open source, includes feature flags)
• Growthbook (open source)
• Custom implementation (as shown above)

Enterprise

• Optimizely, VWO, LaunchDarkly, Split.io

Analysis

• Google Analytics 4, BigQuery, Metabase

Related Skills

• page-cro: For generating test ideas based on CRO principles
• analytics-tracking: For setting up test measurement
• copywriting: For creating variant copy
• product-marketing-context: Foundation context to read first