Growth Experiment

Growth Experiment

When to Use

Use this skill when the user needs to design a rigorous, lean growth experiment with a testable hypothesis, clearly defined metrics, statistical grounding, and a full analysis plan. Specific trigger scenarios:

• User wants to test whether a specific change to a landing page, email flow, onboarding sequence, referral mechanic, or pricing structure will improve a growth metric
• User needs to validate a growth assumption before committing engineering, design, or paid media budget to a full rollout
• User is running a structured growth sprint and needs to prioritize and design experiments from a backlog
• User wants to apply AARRR (Acquisition, Activation, Retention, Revenue, Referral) framework to identify and test a lever in a specific funnel stage
• User has a gut-feel "growth hack" idea and wants to transform it into a testable, repeatable experiment with real success criteria
• User needs to present an experiment brief to a stakeholder, product team, or growth committee for approval
• User wants to instrument a new channel, tactic, or message variation and needs a measurement structure before launching

Do NOT use when:

• User needs a full go-to-market or marketing strategy document -- use marketing-strategy
• User is planning a multi-channel paid campaign with budget allocation and creative direction -- use campaign-planning
• User is designing an in-product feature A/B test with engineering instrumentation and product metrics -- use ab-test-design in product management skills
• User wants to analyze results from an already-completed experiment -- use a data analysis skill
• User is asking for SEO, content, or brand strategy recommendations without a specific testable hypothesis -- use the relevant channel strategy skill
• User wants to run a pricing research study with conjoint analysis or willingness-to-pay surveys -- this requires a research methodology skill

Process

Step 1: Collect Experiment Context

Before writing a single line of the experiment brief, extract the inputs that determine every downstream decision. If the user has not provided these, ask for them explicitly.

• The specific change being tested: One concrete, bounded change -- not "improve onboarding" but "add a progress bar to the 4-step onboarding flow." Vague experiments produce vague learnings.
• AARRR funnel stage: Identify which stage is being targeted -- Acquisition (new visitors/leads), Activation (first value moment), Retention (repeat engagement or churn reduction), Revenue (conversion to paid, upsell), or Referral (invites, word of mouth). This determines the right primary metric class.
• Baseline metric: The current measured rate or value (e.g., "email open rate is 22%", "trial-to-paid conversion is 4.1%", "Day-7 retention is 31%"). Without a baseline, you cannot calculate sample size or detect a meaningful lift.
• Traffic or audience volume: Weekly or monthly volume in the population segment being tested. This directly sets the experiment duration.
• Available resources: Engineering time (can you split-test, or is this a before/after?), design capacity, and timeline constraints. These determine test feasibility.
• Risk tolerance: Is this a low-stakes copy change (low risk) or a pricing or checkout flow change that could depress revenue (high risk)? High-risk experiments require smaller test populations and stop-loss rules.
• Experiment history: Has anything similar been tested before? Previous results reduce uncertainty and tighten the hypothesis.

Step 2: Formulate the Hypothesis

The hypothesis is the intellectual core of the experiment. A weak hypothesis produces an uninterpretable result even if the data is clean.

• Use the canonical form: "If we [specific, single change], then [primary metric] will [increase/decrease] from [baseline] to [target] by [end of experiment duration], because [causal mechanism]."
• The mechanism is mandatory. The "because" clause forces you to commit to a theory of why the change works. Without a mechanism, you cannot learn anything from a negative result or generalize a positive one.
• Target a specific, measurable number. "Conversion rate will increase from 3.2% to 4.0%" is testable. "Conversion rate will improve" is not -- you cannot calculate sample size or define success without a target.
• Write the null hypothesis explicitly. The null is the default assumption: no change, no effect. Being explicit about the null forces rigor and prevents post-hoc rationalization of weak results.
• Check falsifiability. If the experiment cannot possibly produce data that proves the hypothesis wrong, redesign it. Every hypothesis must have a plausible negative outcome.
• Common mistake to avoid: "We believe users want this feature" is not a growth experiment hypothesis -- it is a product discovery hypothesis. A growth experiment hypothesis must target a measurable growth metric (conversion rate, activation rate, LTV, CAC, referral rate).

Step 3: Define the Experiment Structure

Structure the experiment so that the only variable is the change being tested. Contamination and selection bias are the two most common failure modes at this step.

• Control group: The current, unchanged experience. Document it precisely -- take screenshots, export the current email copy, record the current onboarding step count. The control must be reproducible.
• Variant group (or groups): The changed experience. Limit to one primary change per experiment. Testing a new headline and a new CTA simultaneously makes it impossible to attribute causation.
• Randomization method: Random assignment is the gold standard. For web experiments, use a cookie-based or user-ID-based random split. For email experiments, randomly split the list before sending. For paid acquisition experiments, use geographic or time-based splits when user-level randomization is unavailable -- but document the tradeoff.
• Population definition: Define inclusion criteria precisely. "All pricing page visitors" is ambiguous -- specify "all unique visitors to /pricing who have not previously signed up, excluding returning logged-in users." Segment contamination (including users who have already converted) inflates the baseline and masks true lift.
• Holdout groups: For retention and revenue experiments, consider a holdout (a group that never sees the new experience) to measure long-term effects, especially when network effects or habit formation are part of the mechanism.
• Unit of randomization: Always randomize at the user level (user ID or cookie), not the session level. Session-level randomization allows the same user to see both variants, corrupting the experiment.

Step 4: Calculate Sample Size and Duration

This step separates rigorous growth experimentation from cargo-cult testing. Undersized experiments waste time and resources; results are uninterpretable.

• Standard parameters for most growth experiments: 95% confidence level (alpha = 0.05), 80% statistical power (beta = 0.20), two-tailed test. Use one-tailed test only when you have strong prior evidence that the effect cannot be negative.
• Minimum Detectable Effect (MDE): Define the smallest improvement worth implementing. For a pricing page converting at 3.2%, a 0.1% absolute lift (3.1% to 3.2%) may not justify the engineering cost to maintain the variant permanently -- set the MDE at the minimum business-valuable effect (e.g., 0.5% absolute = 15% relative lift).
• Sample size formula (two-proportion z-test): For conversion rates, use n = (Z_alpha/2 + Z_beta)² × [p1(1-p1) + p2(1-p2)] / (p1-p2)². At 95% confidence, 80% power: Z_alpha/2 = 1.96, Z_beta = 0.84. Online calculators (Evan Miller's sample size tool is the standard reference) automate this.
• Practical benchmarks to internalize:
  • A conversion rate of 3% trying to detect a 20% relative lift (to 3.6%) needs ~5,500 users per group
  • A conversion rate of 10% trying to detect a 15% relative lift (to 11.5%) needs ~3,100 users per group
  • A conversion rate of 30% trying to detect a 10% relative lift (to 33%) needs ~1,700 users per group
  • Email open rates (around 20-25%) detecting a 5% absolute lift need ~600 emails per group
• Duration calculation: Divide sample size per group by the weekly volume in that population. Add 10-20% buffer for natural variance. Round up to full weeks to control for day-of-week effects. Never run an experiment for less than 7 days -- weekend vs weekday behavior differences will corrupt results.
• Business cycle awareness: If the product has a monthly billing cycle, a weekly sales rhythm, or a seasonal pattern, the experiment must span at least one full cycle to capture representative behavior.

Step 5: Select the Right Statistical Method

Choosing the wrong statistical test is one of the most common errors in growth experimentation. The test must match the metric type.

• Binary conversion metrics (signup rate, click-through rate, trial-to-paid rate): Two-proportion z-test or chi-squared test. These are the most common growth metrics. The test compares the proportion of successes in control vs variant.
• Continuous metrics (revenue per user, session duration, number of actions): Two-sample t-test (for normally distributed data) or Mann-Whitney U test (for skewed data, which is very common in revenue metrics). Revenue per user is almost always right-skewed -- default to Mann-Whitney or use a log transformation before applying a t-test.
• Bayesian A/B testing: Preferred when you need actionable decisions with smaller sample sizes or when you want to express results as "probability that variant is better" rather than p-values. Tools like VWO, Optimizely, and AB Tasty offer Bayesian modes. Use Bayesian when stakeholders need intuitive probability statements.
• Sequential testing / always-valid inference: Use when you cannot predetermine sample size (continuous traffic) and need to peek at results. Methods like the mSPRT (mixture Sequential Probability Ratio Test) control Type I error even with repeated looks. This is the approach used by Spotify, Airbnb, and Netflix for their internal experimentation platforms.
• For small populations (n < 200 per group): Fisher's Exact Test for binary metrics. Permutation tests for continuous metrics. Avoid chi-squared, which has poor accuracy at small cell counts.

Step 6: Define the Analysis Plan Before Launch

The analysis plan must be locked before any data is collected. Writing it afterward -- even with good intentions -- opens the door to p-hacking.

• Primary metric: One metric. One. Decide what success looks like before you see the data. If you are measuring multiple metrics, apply a Bonferroni correction (divide alpha by the number of metrics) to control the family-wise error rate.
• Secondary metrics: List 3-5 metrics to monitor for unintended effects. These are not decision metrics -- they are guardrail metrics. Example: if the primary metric is signup conversion, guardrail metrics include time-to-first-value, Day-7 retention, and support ticket volume.
• Segmentation plan: Pre-specify 2-3 segments to analyze (e.g., new vs returning visitors, mobile vs desktop, paid vs organic traffic). Post-hoc segmentation fishing is a major source of false positives in growth experimentation. Pre-register the segments.
• No-peeking policy: Set a fixed end date based on the sample size calculation. Do not analyze results until that date unless a stop-loss condition is triggered. If you must monitor for safety (revenue experiments, trust experiments), use sequential testing methods.
• Decision framework: Pre-commit to the four possible outcomes and what you will do in each case: (1) significant positive -- roll out and continue iterating, (2) significant negative -- revert and investigate, (3) inconclusive (underpowered) -- extend experiment or increase traffic, (4) inconclusive (powered but flat) -- file as learned null, do not re-test the same variant.
• Novelty effect check: For UX and email experiments, check whether the lift in the first week is higher than subsequent weeks. Novelty effects inflate results for the variant -- sustained lift over 2+ weeks is more credible than a spike in week one.

Step 7: Apply ICE Prioritization If Multiple Experiments Are in Backlog

Growth teams always have more experiments to run than time to run them. ICE scoring provides a defensible prioritization framework.

• ICE = Impact × Confidence × Ease, each scored 1-10.
• Impact: How large could the effect be if the hypothesis is correct? Anchor to the metric and the funnel stage. An experiment targeting signup conversion at the top of a funnel with high volume has higher maximum impact than one targeting referral from a small existing user base.
• Confidence: How strong is the evidence that this change will work? Prior A/B test results, user research, industry benchmarks, and analogous case studies all increase confidence. A wild guess scores 2-3. A hypothesis supported by user interviews and a competitor's public case study scores 7-8.
• Ease: How fast and cheap is this to implement? A copy change on a landing page scores 9-10. A new referral flow requiring backend engineering scores 2-3.
• ICE threshold for a healthy growth backlog: Run experiments with ICE >= 4.0. Park experiments with ICE < 3.0 for reconsideration after 2 sprint cycles.
• Practical use: If presenting a growth experiment design alongside a backlog, include a mini-ICE scorecard to justify why this experiment is being run now.

Step 8: Write the Experiment Brief and Risk Plan

The brief is the artifact that aligns the team, prevents scope creep, and creates institutional memory.

• One-page summary: Hypothesis, primary metric, sample size, duration, and success criteria. Stakeholders should be able to understand the experiment in 60 seconds.
• Implementation checklist: Tag setup in analytics (Google Analytics 4 event, Mixpanel funnel, Segment event), QA check on variant rendering across device types, URL parameter or cookie logic verified, holdout group confirmed.
• Stop-loss triggers: For revenue and trust experiments, define an automatic stop condition. Example: "If the variant reduces trial-to-paid conversion below 2.5% (the current 4.1% rate minus a 40% drop) at any point during the experiment, pause immediately and revert."
• Post-experiment documentation: Commit to writing a results card regardless of outcome. The results card includes the final metric data, statistical test result, interpretation, and recommended next action. Negative results are as valuable as positive ones -- a null result rules out a class of interventions and prevents future teams from re-testing the same thing.

Output Format

## Growth Experiment: [Experiment Name]

**Experiment ID:** [EXP-YYYY-NNN for experiment log tracking]
**Growth Stage:** [Acquisition / Activation / Retention / Revenue / Referral]
**Funnel Position:** [Specific funnel step, e.g., "Pricing page → Signup"]
**Owner:** [Name or role]
**Date Designed:** [Date]
**Planned Launch:** [Date]
**Planned End Date:** [Date]
**Status:** [Designed / Running / Complete / Paused]
**ICE Score:** [Impact: X / Confidence: X / Ease: X / Total: X.X]

---

### Hypothesis

**If** we [specific, single, bounded change -- be precise about placement, copy, mechanic, or flow],
**then** [primary metric] will [increase / decrease] from [current baseline] to [target value] ([relative % change]),
**by** [experiment end date],
**because** [causal mechanism -- the theory of why this change moves the metric].

**Null Hypothesis:** [The change has no measurable effect on [primary metric]. The observed rate will remain within normal variance of the [baseline] baseline.]

**Assumed Mechanism:** [1-2 sentences explaining the behavioral or psychological theory behind the change. E.g., "Social proof from specific customers reduces decision uncertainty at the moment of highest cognitive load, lowering the perceived risk of commitment."]

---

### Experiment Design

| Component | Detail |
|-----------|--------|
| Control | [Current state -- what users experience without the change. Be specific.] |
| Variant | [The single change being tested. Include exact copy, placement, or mechanic.] |
| Primary Metric | [One metric. Include definition: numerator / denominator.] |
| Guardrail Metrics | [2-4 metrics to monitor for unintended harm. Not decision metrics.] |
| Population | [Who is included. Inclusion and exclusion criteria. Funnel stage.] |
| Randomization Unit | [User ID / Cookie / Email subscriber ID -- user-level, not session-level] |
| Traffic Split | [50/50 or stated split with justification for asymmetric splits] |
| Sample Size | [N per group, with method: "2,500/group based on two-proportion z-test, MDE = 0.8% absolute, 95% CI, 80% power"] |
| Duration | [Weeks. Include: daily/weekly volume, calculation, and business cycle check] |
| Confidence Level | [95% (standard) or stated alternative with justification] |
| Statistical Test | [Two-proportion z-test / t-test / Mann-Whitney U / Bayesian / Sequential] |

---

### Minimum Detectable Effect (MDE) Calculation

| Parameter | Value |
|-----------|-------|
| Baseline Rate | [e.g., 3.2%] |
| Target Rate (MDE) | [e.g., 4.0%] |
| Absolute Lift | [e.g., +0.8 percentage points] |
| Relative Lift | [e.g., +25%] |
| Business Justification for MDE | [Why is this the minimum worth implementing? e.g., "At current traffic, each 0.1% lift = 5 additional signups/month. 0.8% lift = 40 signups/month = $X in pipeline at current close rate."] |

---

### Success Criteria

| Outcome | Statistical Condition | Business Condition | Action |
|---------|----------------------|-------------------|--------|
| Positive | p < 0.05 (or P(variant > control) > 95% Bayesian) | Primary metric >= [target] | Roll out to 100% of traffic. Document. Plan follow-on iteration. |
| Marginal Positive | p < 0.05 but effect < MDE | Primary metric between baseline and [target] | Decision meeting: is the effect worth implementation cost? Do not auto-roll-out. |
| Inconclusive -- Underpowered | p >= 0.05, experiment ended early or traffic lower than expected | N/A | Extend experiment by [X weeks] or identify traffic acceleration. |
| Inconclusive -- Null | p >= 0.05, full sample collected | Metric change < 0.2% absolute | File as null result. Hypothesis mechanism disproved. Do not re-test this variant. |
| Negative | p < 0.05 (variant worse than control) | Primary metric < baseline | Revert immediately. Investigate mechanism. Document learnings. |

---

### Stop-Loss Rules

| Trigger | Threshold | Action |
|---------|-----------|--------|
| Revenue / Conversion Drop | [Primary metric falls below baseline - 20% at any point] | Pause experiment, revert variant, convene post-mortem |
| Error Rate Spike | [JavaScript errors or 404s exceed 0.5% of variant sessions] | Pause experiment, fix implementation, restart with clean data |
| Sample Ratio Mismatch | [Actual split deviates from 50/50 by more than 5 percentage points] | Pause. Investigate randomization bug. Do not analyze results until resolved. |

---

### Implementation Plan

| # | Step | Owner | Deadline | Deliverable | Done? |
|---|------|-------|----------|-------------|-------|
| 1 | [Analytics instrumentation -- define events, confirm tracking fires correctly] | [Owner] | [Date] | QA-confirmed event schema |
| 2 | [Variant build -- design and develop the change] | [Owner] | [Date] | Staging environment ready |
| 3 | [QA -- test variant on all target devices and browsers] | [Owner] | [Date] | QA sign-off |
| 4 | [Launch -- enable experiment for defined traffic split] | [Owner] | [Date] | Experiment live, baseline data confirmed |
| 5 | [Mid-point check -- guardrail metrics only, no primary metric peek] | [Owner] | [Date + half duration] | Guardrail status: green / amber / red |
| 6 | [Analysis -- run statistical test, write results card] | [Owner] | [End date + 2 days] | Results card posted to experiment log |
| 7 | [Decision + rollout / revert] | [Owner] | [End date + 5 days] | Implementation decision documented |

---

### Risk Register

| Risk | Probability | Impact | Prevention | Response |
|------|-------------|--------|-----------|----------|
| [Sample ratio mismatch due to cookie deletion] | Medium | High (invalidates results) | Verify randomization logic in QA; monitor split ratio daily | Pause, debug, restart |
| [Novelty effect inflates early results] | Medium | Medium (false positive) | Run experiment for minimum 2 full weeks | Do not call results early; check week-over-week trend |
| [External event contaminates results (product launch, PR hit)] | Low | High | Note all external events in experiment log | Segment out affected time period if large enough; extend experiment |
| [Variant negatively affects a segment not in the primary metric] | Low | High | Monitor guardrail metrics; segment results by user type | Stop-loss if guardrail triggers |

---

### Analysis Plan

**Primary Statistical Test:** [Method and tool -- e.g., "Two-proportion z-test using Python scipy.stats.proportions_ztest or Evan Miller's A/B testing calculator"]
**Significance Threshold:** alpha = 0.05 (p < 0.05 required for positive decision)
**Power:** 80% (beta = 0.20)
**Test Type:** [Two-tailed (default) or one-tailed with justification]

**Pre-registered Segments for Analysis:**
1. [e.g., New visitors vs returning visitors]
2. [e.g., Mobile vs desktop]
3. [e.g., Organic vs paid traffic source]

**Guardrail Metric Thresholds:**
- [Guardrail 1]: Alert if [metric] changes by more than [threshold] in the variant
- [Guardrail 2]: Alert if [metric] changes by more than [threshold] in the variant

**Peeking Policy:** Results will not be analyzed before [end date]. The only exception is a stop-loss trigger. If you check results early, you must apply a Bonferroni correction (divide alpha by number of peeks) or use sequential testing methods.

**Novelty Effect Check:** Compare week 1 conversion rate vs weeks 2+ in the variant. If week 1 is more than 20% higher than subsequent weeks, flag as potential novelty effect and extend experiment.

**Results Documentation:** Complete a results card within 48 hours of experiment end, regardless of outcome. File in [experiment log location].

---

### Results Card (to be completed post-experiment)

| Field | Value |
|-------|-------|
| Final Control Rate | [X%] |
| Final Variant Rate | [X%] |
| Absolute Lift | [+/- X percentage points] |
| Relative Lift | [+/- X%] |
| p-value | [X.XX] |
| Statistical Significance | [Yes / No] |
| Sample Size Achieved | [N per group] |
| Guardrail Metrics Status | [All green / Amber: describe / Red: describe] |
| Decision | [Roll out / Revert / Extend / Null] |
| Key Learning | [1-2 sentences on what this result tells you about the mechanism] |
| Next Experiment | [What to test next based on this result] |

Rules

1. The hypothesis must contain a mechanism. A prediction without a causal theory ("adding testimonials will increase conversion because social proof reduces purchase anxiety") is a hypothesis. A prediction without a mechanism ("adding testimonials will increase conversion") is a wish. Without a mechanism, a negative result teaches you nothing and a positive result cannot be generalized.

2. Set success criteria before the experiment launches -- never after. Post-hoc success criteria are p-hacking by another name. If you define success as "any statistically significant improvement" after seeing results, you will find something to celebrate in noise. The MDE and decision thresholds must be documented and agreed upon before any data is collected.

3. Calculate sample size for every experiment, even rough ones. An experiment that ends with 200 users per group when the MDE required 2,000 per group is not an experiment -- it is anecdote. If you cannot reach statistical significance, either acknowledge the weaker evidence level explicitly or extend the timeline until you can.

4. Randomize at the user level, not the session level. Session-level randomization allows the same user to see both control and variant, which violates the independence assumption of every standard statistical test and biases results toward the null. Always use user ID or persistent cookie for the randomization unit.

5. Watch for the sample ratio mismatch (SRM). If your 50/50 split produces 3,000 users in control and 2,200 in variant, your randomization is broken -- something is systematically routing users differently between the groups. An SRM invalidates the experiment. Check the actual split ratio within the first 24-48 hours and do not analyze results from an SRM-corrupted experiment.

6. Run every experiment for at least 7 full days. Day-of-week behavior patterns (weekday vs weekend intent, B2B Monday peak, consumer Friday engagement) will corrupt experiments shorter than 7 days. Experiments running over monthly boundaries should account for billing cycle effects. Two full weeks is the practical minimum for most growth experiments.

7. One primary metric per experiment. If you measure 10 metrics and call whichever one reaches significance a win, you are running 10 experiments with a 5% false positive rate each -- the probability of at least one false positive is 40%. If you must measure multiple outcomes, apply Bonferroni correction (alpha / number of tests) or pre-register a clear primary metric hierarchy.

8. Do not run overlapping experiments on the same population without mutual exclusion or interaction analysis. Two experiments running concurrently on the same users contaminate each other. If experiment A changes the email subject line and experiment B changes the email CTA at the same time, and both show lift, you cannot attribute which variable drove what. Use a testing platform with mutual exclusion support or run experiments sequentially on the same population.

9. Revenue and pricing experiments require a stop-loss trigger and a smaller initial rollout. A bad pricing experiment can destroy days or weeks of revenue. Set an automatic stop condition before launch (e.g., "if trial-to-paid conversion drops below 2.5% in the variant at any point, pause immediately") and start with 10-20% of traffic in the variant rather than 50%.

10. Document every result -- especially null and negative results. The most expensive growth mistake is re-testing a hypothesis that has already been tested and failed. A searchable experiment log with results cards for every experiment (positive, null, negative) is the primary asset of a mature growth function. Over 24 months, a team running 2-3 experiments per week accumulates 150-200 results -- this is a competitive advantage only if it is searchable and readable.

Edge Cases

Small User Base (Under 1,000 Monthly Users in the Target Funnel Segment)

Statistical significance is unreachable in reasonable timeframes. Do not pretend otherwise. Instead: set a practical significance threshold based on the minimum effect size that would change a business decision (e.g., "if conversion doubles from 5% to 10%, we invest; if it doesn't, we don't"). Run the experiment for a fixed period (typically 4-8 weeks), report the observed effect and confidence interval, and be explicit that the result is directional rather than conclusive. Supplement quantitative data with 5-10 user interviews and session recording analysis (tools like FullStory or Hotjar). Treat the result as a strong hypothesis to re-test at larger scale, not a decision-ready finding.

No Engineering Resources for A/B Testing (Before/After or Geographic Split)

A concurrent randomized controlled trial is the gold standard. If it is unavailable, use the strongest feasible design and document the limitation explicitly. For before/after tests: choose a baseline period that matches the test period in length, seasonality, and external conditions. Control for known confounders (e.g., if a product launch happened mid-baseline, exclude that window). For geographic splits (e.g., run the experiment in one city or country, use another as control): confirm the populations are comparable on key dimensions (demographics, product maturity, pricing). Label the resulting evidence as "quasi-experimental" and recommend a proper RCT when resources allow.

Retention Experiments (Long Time Horizon Required for True Measurement)

True retention effects often take 30-90 days to materialize, which is incompatible with weekly sprint cycles. Solution: identify a leading indicator metric that is both faster to measure and validated as predictive of long-term retention. Common validated leading indicators include: Feature X used within first 7 days predicting 30-day retention (the "activation metric"), number of connections or content items created in week 1 predicting 90-day retention, and email engagement in the first 2 weeks predicting subscription renewal. Use the leading indicator as the primary metric for speed, but commit to measuring the actual retention outcome at the full horizon and updating the results card. Never close a retention experiment loop without eventually measuring the metric you actually care about.

Pricing Experiments

The most risk-laden category of growth experiment. A 20% increase in pricing that reduces conversion by 30% is not just a failed experiment -- it is lost revenue for the duration of the test. Mandatory constraints: (1) Limit variant exposure to 10-20% of new users (never existing customers); (2) Set a hard stop-loss threshold before launch; (3) Never show different prices to users who might compare notes (support forums, communities, B2B buying teams); (4) Ensure legal and compliance review if your pricing experiment involves different prices for protected classes of users. Measure not just conversion rate but revenue per visitor and LTV. A variant that converts fewer users but at higher price may be net positive -- you need revenue metrics, not just conversion metrics.

Experiment Contamination from External Events

A product launch, a viral PR event, a competitor's announcement, or a paid campaign spike during the experiment period can contaminate results by changing the composition of the audience or the baseline intent level. Prevention: log all external events in the experiment record from day one. Detection: monitor the control group's primary metric daily -- if the control group's rate changes significantly mid-experiment, something external has happened. Response: if the contamination is time-bounded (e.g., a 3-day PR spike), segment out the affected period and re-run the analysis on the clean window. If contamination is pervasive, extend the experiment until you have a clean equivalent period in both arms.

Multi-Variant Experiments (Testing 3+ Variants Simultaneously)

Testing multiple variants (e.g., three different CTA copy options against a control) is tempting because it seems efficient. The statistical cost is real: with 4 groups (1 control + 3 variants), you need 4x the total sample size and must apply multiple comparison corrections. For growth experiments, limit to 1-2 variants unless you have a specific reason (e.g., you are testing categorically different approaches and have the traffic to support it). If you run 3+ variants, use a false discovery rate (FDR) correction method like the Benjamini-Hochberg procedure rather than Bonferroni, which is overly conservative for multiple comparisons. Pre-specify which variant you will roll out if multiple variants beat control -- do not pick the winner opportunistically after seeing data.

Experiments with Network Effects or Viral Mechanics

If the experiment tests a referral mechanic, a social sharing feature, or any viral loop, standard user-level randomization breaks down because the treatment effect on one user spills over to others in the same network. A user in the control group who receives an invite from a user in the variant group is contaminated. This is the interference problem, also called SUTVA (Stable Unit Treatment Value Assumption) violation. Solutions include: cluster randomization (randomize by social graph cluster or geographic market, not individual user), ego-network analysis, or switchback experiments (time-based, not user-based randomization). Acknowledge the limitation explicitly if none of these approaches is feasible and interpret results as lower bounds on the true effect.

Example

Input: "We run a B2B SaaS product for project management. We want to test whether adding an in-app tooltip prompt during the onboarding flow -- specifically on the 'Create your first project' step -- will improve our Week-1 activation rate. Currently, only 38% of new signups create a first project within 7 days. We get about 800 new signups per month. The team can implement this in a week."

Output:

Growth Experiment: In-App Tooltip on 'Create First Project' Step

Experiment ID: EXP-2024-017 Growth Stage: Activation -- First Value Moment (Signup → First Project Created Within 7 Days) Funnel Position: Onboarding Step 3 "Create Your First Project" → Day-7 Activation Owner: Growth Lead Date Designed: [Today] Planned Launch: [Today + 7 days (implementation window)] Planned End Date: [Launch date + 35 days] Status: Designed ICE Score: Impact: 8 / Confidence: 7 / Ease: 8 / Total: 7.7

Hypothesis

If we add a contextual tooltip on the "Create Your First Project" onboarding step that displays a pre-filled project template with three example tasks and a one-click "Start with this template" button, then the percentage of new signups who create at least one project within 7 days of signup (Week-1 activation rate) will increase from 38% to 46% or higher (+8 percentage points absolute, +21% relative improvement), by the end of a 35-day experiment window, because the primary barrier to first project creation is the blank-canvas problem -- new users do not know where to start, experience decision paralysis when faced with an empty project view, and abandon before experiencing the core value of the product. A pre-filled template with realistic content reduces this friction by giving users a concrete starting point that requires only minimal customization rather than a creation effort from scratch.

Null Hypothesis: Adding a tooltip with a pre-filled template has no measurable effect on Week-1 activation rate. The observed activation rate in the variant group will remain within normal variance of the 38% baseline.

Assumed Mechanism: Activation failures at this step are predominantly effort-and-uncertainty driven, not motivation driven. Users who reach "Create First Project" have already committed to trying the product -- the dropout is caused by implementation friction, not intent. A template lowers the effort threshold from "create something meaningful" to "edit something that already exists," which is a well-established cognitive load reduction technique.

Experiment Design

Minimum Detectable Effect (MDE) Calculation

Success Criteria

Stop-Loss Rules

…(truncated)