Tpl Situacao Ab Testing Implementacao

SITUATION: A/B Testing Implementation

Skill gerado a partir do pack templates-claude-code. Arquivo de origem: situacao/17-ab-testing-implementacao.md. Use como baseline e adapte ao projeto antes de mudancas grandes.

Conteudo do template

CONTEXT

Use this configuration when implementing A/B tests (split tests), multivariate tests, or feature flag-driven experiments. A/B testing is not just a marketing tool — it's an engineering discipline. Badly implemented experiments produce misleading data that leads to wrong product decisions. This configuration ensures statistical rigor, implementation correctness, and clean winner deployment.

OBJECTIVES

• Run experiments that produce statistically valid and actionable results
• Prevent experiment pollution (interactions between experiments)
• Ensure consistent user assignment across sessions (sticky bucketing)
• Define success metrics and minimum sample size before starting
• Deploy winning variants cleanly and remove experiment code when done

APPROACH RULES

1. One variable at a time. An A/B test must change exactly one variable. Changing button color AND copy in the same test makes it impossible to attribute the result. If you need to test multiple things, run sequential or multivariate tests with proper design.

2. Define the success metric before starting. What exactly will you measure? What minimum effect size would justify shipping? What sample size do you need to detect it? These answers must exist before traffic is split. Post-hoc metric selection is p-hacking.

3. Calculate minimum sample size before starting. Use a power calculator. Typical parameters: 80% statistical power, 95% confidence level, effect size based on minimum meaningful change. Never end an experiment early because "it looks good."

4. Sticky bucketing is non-negotiable. A user assigned to variant A must always see variant A. Assignment must be deterministic (hash of user ID + experiment ID), not random per request. Changing assignment mid-experiment invalidates the data.

5. Guard against novelty effect. New UI elements always get more clicks initially due to novelty. Run experiments for at least one full business cycle (usually 1-2 weeks minimum) to account for day-of-week effects and novelty decay.

6. Stop experiments only when sample size is reached or for safety. Do not stop when you see a positive result "too early." Do not keep running indefinitely. Automated stopping rules (sequential testing) are valid only if configured before the experiment starts.

7. Clean up experiment code. A winner means: remove the losing variant, remove the feature flag, promote the winning code to be the permanent implementation. Technical debt from undead experiments compounds quickly.

ROUTING TABLE

Experiment Design Template

## Experiment: [Name]

**Hypothesis:**
If we [change X], then [metric Y] will [increase/decrease] by [Z%],
because [user psychology/behavior reason].

**Variants:**
- Control (A): [Current behavior - describe precisely]
- Variant B: [What changes - describe precisely]

**Primary Metric:** [One metric. If you have multiple, pick the most important.]
- Checkout completion rate

**Secondary Metrics (monitoring, not decision):**
- Revenue per visitor
- Bounce rate
- Time on page

**Guardrail Metrics (must not degrade):**
- Site error rate
- Page load time P95

**Minimum Detectable Effect:** 5% relative improvement
**Statistical Power:** 80%
**Confidence Level:** 95%
**Required Sample Size:** 12,400 users per variant (calculated via power analysis)
**Estimated Duration:** 10 days at current traffic volume

**Traffic Split:** 50% control / 50% variant
**Targeting:** All logged-in users on checkout page

**Assignment:** hash(user_id + "experiment_checkout_cta_v2") % 100 < 50 → control

**Start Date:** [date]
**Minimum End Date:** [start + duration. Never end before this.]
**Decision Date:** [start + duration + 2 days analysis buffer]

Feature Flag Architecture

interface ExperimentAssignment {
  variant: 'control' | 'treatment'
  experimentId: string
  userId: string
  assignedAt: string
}

class ExperimentService {
  /**
   * Deterministic, sticky assignment using user ID hash.
   * Same user always gets same variant for same experiment.
   */
  getVariant(userId: string, experimentId: string): 'control' | 'treatment' {
    // Hash must be consistent — not time-dependent
    const hash = murmurhash(`${userId}:${experimentId}`)
    const bucket = hash % 100
    
    const experiment = this.getExperiment(experimentId)
    
    if (!experiment || !experiment.isActive) {
      return 'control' // Default: show control when no experiment
    }
    
    return bucket < experiment.treatmentPercentage ? 'treatment' : 'control'
  }

  /** Always log assignments for analysis */
  assignAndLog(userId: string, experimentId: string): ExperimentAssignment {
    const variant = this.getVariant(userId, experimentId)
    
    const assignment: ExperimentAssignment = {
      variant,
      experimentId,
      userId,
      assignedAt: new Date().toISOString()
    }
    
    // Log once per user per experiment (not every request)
    if (!this.hasLoggedAssignment(userId, experimentId)) {
      this.analyticsClient.track('experiment_enrolled', assignment)
      this.cacheAssignment(userId, experimentId)
    }
    
    return assignment
  }
}

Statistical Analysis

from scipy import stats
import numpy as np

def analyze_ab_test(
    control_conversions: int, 
    control_visitors: int,
    treatment_conversions: int,
    treatment_visitors: int,
    alpha: float = 0.05
) -> dict:
    """Returns analysis result with clear ship/no-ship recommendation."""
    
    control_rate = control_conversions / control_visitors
    treatment_rate = treatment_conversions / treatment_visitors
    relative_lift = (treatment_rate - control_rate) / control_rate
    
    # Two-proportion z-test
    stat, p_value = stats.proportions_ztest(
        [treatment_conversions, control_conversions],
        [treatment_visitors, control_visitors]
    )
    
    is_significant = p_value < alpha
    
    return {
        'control_rate': f"{control_rate:.2%}",
        'treatment_rate': f"{treatment_rate:.2%}",
        'relative_lift': f"{relative_lift:+.1%}",
        'p_value': round(p_value, 4),
        'is_significant': is_significant,
        'recommendation': 'SHIP' if (is_significant and relative_lift > 0) else 'NO_SHIP',
        'confidence': f"{(1-p_value)*100:.1f}%"
    }

DO NOT

• DO NOT analyze results before minimum sample size is reached
• DO NOT change the primary metric after the experiment starts
• DO NOT run an experiment on < 1 week of data (day-of-week effects)
• DO NOT make a decision based on secondary metrics if the primary metric is flat
• DO NOT leave losing variant code in the codebase after concluding the experiment
• DO NOT count the same user's sessions as independent samples
• DO NOT run experiments on new or infrequent users only (survivorship bias)
• DO NOT include bot/crawler traffic in experiment analysis

OUTPUT FORMAT

At experiment conclusion, produce an Experiment Report:

## Experiment Report: [Name]

**Duration:** 2024-01-01 to 2024-01-14 (14 days)
**Total Visitors:** 28,400 (14,200 per variant)

### Results

| Variant | Visitors | Conversions | Rate | vs Control |
|---------|----------|-------------|------|-----------|
| Control | 14,200 | 852 | 6.00% | — |
| Treatment | 14,200 | 924 | 6.51% | +8.4% |

**p-value:** 0.023 (statistically significant at 95% confidence)
**Relative Lift:** +8.4% on checkout completion rate

### Decision: **SHIP TREATMENT** ✅

### Next Steps
- [ ] Deploy treatment as permanent implementation
- [ ] Remove control variant code
- [ ] Archive experiment config (keep 90 days for reference)
- [ ] Remove feature flag
- [ ] Document learnings in team wiki

QUALITY GATES

• Hypothesis written before experiment starts
• Primary metric defined before experiment starts
• Sample size calculated with power analysis before starting
• Minimum experiment duration respected (no early stopping except safety)
• Sticky bucketing verified: same user always gets same variant
• Bot traffic excluded from analysis
• One primary metric — not "any of these 5 look good"
• Both losing variant code AND feature flag removed after conclusion
• Experiment report documented and accessible to product/leadership
• Guardrail metrics checked: no degradation in error rate or performance