majiayu000 Growth Experimenter
Run systematic experiments to grow faster through data-driven optimization.
Core Philosophy
Growth = Experimentation Velocity × Win Rate × Impact per Win
- Run more experiments
- Increase your hit rate through better hypotheses
- Focus on high-impact areas
Growth Model (AARRR / Pirate Metrics)
Acquisition → Activation → Retention → Revenue → Referral
↓ ↓ ↓ ↓ ↓
Traffic Sign Up Day 30 Upgrade Invites
100% 40% 50% 20% 10%
Example: 10,000 visitors/month
→ 4,000 signups (40%)
→ 2,000 active at D30 (50%)
→ 400 paying (20%)
→ 40 referrals (10%)
Improve ANY metric by 10% = 10% more customers
Where to focus first: The leakiest bucket
- If 40% sign up but only 10% are active at D30 → Fix retention
- If 80% are active but only 5% pay → Fix monetization
- If 2% visitors sign up but 60% convert to paid → Get more traffic
Experiment Framework
1. Identify the Problem
Good problem statements:
- "Only 2% of homepage visitors sign up" (specific metric)
- "50% of trials don't complete onboarding" (clear drop-off)
- "Users who invite teammates have 3x retention, but only 10% invite" (known behavior)
Bad problem statements:
- "We need more growth" (too vague)
- "Conversion is bad" (no baseline)
- "Users don't understand the product" (not measurable)
2. Form a Hypothesis
Hypothesis template:
We believe that [change]
will result in [outcome]
because [reason/evidence]
Examples:
✅ Good:
We believe that adding social proof (testimonials) to the pricing page
will increase trial signups by 10%
because visitors currently have low trust and need validation.
✅ Good:
We believe that sending a Slack notification when user completes setup
will increase D7 activation by 20%
because users forget to come back after initial signup.
❌ Bad:
We believe that changing the button color will improve conversions
(no reason why)
❌ Bad:
We believe that improving the product will increase retention
(too vague, not testable)
3. Design the Experiment
Experiment specification:
Experiment: Add social proof to pricing page
Hypothesis: Social proof on pricing will increase signups by 10%
Variants:
Control: Current pricing page (no testimonials)
Treatment: Pricing page + 3 customer testimonials
Primary Metric: Trial signup rate
Secondary Metrics:
- Time on page
- Scroll depth
- CTA click rate
Sample Size: 1,000 visitors per variant
Duration: 2 weeks (or until statistical significance)
Success Criteria: >5% improvement with 95% confidence
Measurement:
- Google Analytics
- Mixpanel conversion tracking
- Segment for event data
4. Run the Experiment
A/B testing checklist:
- Random assignment (50/50 split)
- Same time period (no day-of-week effects)
- Sufficient sample size
- No peeking (wait for significance)
- One change at a time
Statistical significance calculator:
// Minimum sample size for 95% confidence
function calculateSampleSize(baseline, mde, power = 0.8, alpha = 0.05) {
// baseline = current conversion rate (e.g., 0.02)
// mde = minimum detectable effect (e.g., 0.10 for 10% lift)
// Returns: visitors needed per variant
const z_alpha = 1.96 // 95% confidence
const z_power = 0.84 // 80% power
const p1 = baseline
const p2 = baseline * (1 + mde)
const p_avg = (p1 + p2) / 2
const n = (2 * p_avg * (1 - p_avg) * (z_alpha + z_power) ** 2) / (p2 - p1) ** 2
return Math.ceil(n)
}
// Example: 2% baseline, detect 10% improvement
calculateSampleSize(0.02, 0.1) // ~35,000 visitors per variant
5. Analyze Results
Interpreting results:
Control: 1,000 visitors → 20 conversions (2.0%)
Treatment: 1,000 visitors → 25 conversions (2.5%)
Lift: +25% relative (+0.5% absolute)
P-value: 0.04 (statistically significant if <0.05)
Confidence Interval: [-0.2%, +1.2%]
Decision: WIN - Ship it!
When results are inconclusive:
- No movement: Hypothesis was wrong or change too small
- Not significant: Need more data or larger effect
- Negative impact: Roll back immediately
- Contradictory secondary metrics: Investigate trade-offs
6. Scale What Works
// After successful experiment, roll out to 100%
if (experimentResult.lift > 0.05 && experimentResult.pValue < 0.05) {
rolloutFeature({
feature: 'social_proof_on_pricing',
rollout: '100%',
monitor: ['signup_rate', 'trial_starts']
})
// Log the learning
logExperimentLearning({
learning: 'Social proof increased signups by 25%',
application: 'Add social proof to all high-intent pages'
})
}
Growth Experiments by Stage
Acquisition Experiments
Goal: Get more traffic or improve traffic quality
High-impact experiments:
- Landing page optimization:
Control: Generic homepage
Test: Tailored landing pages by traffic source
- /for-startups (Product Hunt traffic)
- /for-agencies (Google Ads)
- /for-developers (GitHub referrals)
Expected lift: 20-50% on signup rate
- Headline testing:
Current: 'Project Management Software'
Test A: 'Ship Projects 2x Faster'
Test B: 'The Project Management Tool Teams Love'
Test C: "Finally, Project Management That Doesn't Suck"
Test: Value prop clarity, specificity, emotion
Expected lift: 10-30% on engagement
- Social proof:
Current: No social proof
Test: Add testimonials, logos, user count
- "Join 10,000+ teams..."
- Customer logos (recognizable brands)
- Video testimonial from power user
Expected lift: 15-25% on trust/signups
Activation Experiments
Goal: Get users to "aha moment" faster
High-impact experiments:
- Onboarding simplification:
Current: 7-step onboarding flow
Test: 3-step flow, delay advanced setup
Step 1: Name + email
Step 2: Create first project
Step 3: Invite team (optional, skippable)
Expected lift: 30-50% completion rate
- Time-to-value reduction:
Current: Users must create project from scratch
Test: Pre-populated template
- Sample project with tasks
- Example data to explore
- Guided tutorial
Expected lift: 25-40% in D1 activation
- Progress indicators:
Current: No feedback during setup
Test: Progress bar + completion checklist
[✓] Account created
[✓] First project
[ ] Invite teammates (2 left)
[ ] Complete first task
Expected lift: 15-25% completion rate
Retention Experiments
Goal: Keep users coming back
High-impact experiments:
- Email re-engagement:
Current: No emails after signup
Test: 3-email onboarding sequence
Day 1: "Here's how to get started"
Day 3: "Tips from power users"
Day 7: "You're only 1 step away from [value]"
Expected lift: 20-35% in D30 retention
- Habit building:
Current: No reminders
Test: Daily digest email
"Your daily update: 3 tasks due today"
- Creates daily habit
- Drives return visits
Expected lift: 25-40% in daily active users
- Feature discovery:
Current: All features visible, overwhelming
Test: Progressive disclosure
- Week 1: Core features only
- Week 2: Unlock integrations
- Week 3: Unlock advanced features
- Tooltip hints for new features
Expected lift: 15-25% feature adoption
Revenue Experiments
Goal: Convert free users to paying customers
High-impact experiments:
- Paywall optimization:
Current: Hard limit at 5 projects
Test: Soft limit + banner
"You've created 5 projects! Upgrade to Pro for unlimited"
- Allow them to continue
- Show banner on every page
- Show upgrade modal on 6th project
Expected lift: 20-30% in upgrade rate
- Trial length:
Current: 14-day trial
Test A: 7-day trial (more urgency)
Test B: 30-day trial (more time to get hooked)
Test C: Usage-based trial (100 tasks)
Expected: Depends on product complexity
- Pricing page:
Current: 3 tiers without highlight
Test: Highlight "Most Popular" tier
- Green border
- "Most popular" badge
- Slightly larger
Expected lift: 10-20% on middle tier selection
Referral Experiments
Goal: Turn users into advocates
High-impact experiments:
- Invite mechanics:
Current: "Invite" link in settings
Test: Contextual invite prompts
- After completing first task: "Invite your team to help!"
- When tagging someone: "user@example.com isn't on your team yet. Invite them?"
Expected lift: 50-100% in invites sent
- Referral incentives:
Current: No incentive
Test: Double-sided reward
- Referrer: 1 month free
- Referred: 20% off first year
- Must convert to paid
Expected lift: 30-50% in referred signups
- Public profiles:
Current: All projects private
Test: Optional public project sharing
- "Made with [Product]" badge
- Share project publicly
- View-only link with signup CTA
Expected lift: 10-20% referred traffic
Advanced Techniques
Sequential Testing
When traffic is low, use sequential testing instead of fixed-sample A/B:
def sequential_test(control_conversions, control_visitors,
test_conversions, test_visitors):
"""
Evaluate experiment continuously instead of waiting for sample size.
Stop early if clear winner or clear loser.
"""
log_likelihood_ratio = calculate_llr(
control_conversions, control_visitors,
test_conversions, test_visitors
)
if log_likelihood_ratio > 2.996: # 95% confidence winner
return "WINNER"
elif log_likelihood_ratio < -2.996: # 95% confidence loser
return "LOSER"
else:
return "CONTINUE"
Multi-Armed Bandit
Automatically allocate more traffic to winning variants:
class MultiArmedBandit:
def select_variant(self, variants):
"""
Thompson Sampling:
- Start with equal probability
- As data comes in, shift traffic to winners
- Explore new variants occasionally
"""
samples = []
for v in variants:
# Sample from beta distribution
sample = np.random.beta(
v.successes + 1,
v.failures + 1
)
samples.append(sample)
return variants[np.argmax(samples)]
Cohort Analysis
Segment results by user attributes:
Overall lift: +10%
By segment:
Mobile users: +25% (big win!)
Desktop users: +2% (no effect)
Organic traffic: +30% (huge!)
Paid traffic: -5% (negative!)
Action: Roll out to mobile + organic only
North Star Metric
Define one metric that represents customer value:
Examples:
Slack: Weekly Active Users (WAU)
Airbnb: Nights Booked
Facebook: Daily Active Users (DAU)
Spotify: Time Listening
Shopify: GMV (Gross Merchandise Value)
Your North Star should: ✅ Correlate with revenue
✅ Measure value delivery
✅ Be measurable frequently
✅ Rally the entire team
Experiment Ideas Library
Quick Wins (1 week effort)
1. Homepage CTA text: "Start Free Trial" vs "Get Started Free"
2. Signup button color: Blue vs Green vs Red
3. Email subject lines: A/B test 2 variations
4. Pricing page order: Starter-Pro-Business vs Business-Pro-Starter
5. Social proof location: Above fold vs below fold
Medium Effort (2-4 weeks)
1. Redesign onboarding flow (reduce steps)
2. Add email drip campaign
3. Create upgrade prompts in-app
4. Build referral program
5. Redesign pricing page
Big Bets (1-3 months)
1. Launch freemium model
2. Build marketplace/app store
3. Add AI-powered features
4. Redesign entire product (better UX)
5. Build mobile apps
Experiment Tracking
Document Every Experiment
Experiment Log:
Exp-001:
Name: Add social proof to homepage
Start Date: 2024-01-15
End Date: 2024-02-01
Status: ✅ WIN
Hypothesis: Social proof will increase signups by 10%
Result: +18% signup rate, p=0.02
Learnings: Customer logos work better than testimonials
Actions: Roll out to 100%, add logos to pricing page too
Exp-002:
Name: 7-day trial instead of 14-day
Start Date: 2024-02-05
Status: ❌ LOSS
Hypothesis: Shorter trial creates urgency
Result: -12% trial-to-paid conversion, p=0.01
Learnings: Users need more time to integrate product
Actions: Keep 14-day trial, don't test shorter
Exp-003:
Name: Onboarding simplification
Start Date: 2024-02-15
Status: ⏳ RUNNING
Hypothesis: 3-step flow will improve completion by 30%
Current: +22% completion, n=850, p=0.08 (not yet significant)
Experiment Prioritization
ICE Score Framework:
Impact (1-10): How much could this move the needle?
Confidence (1-10): How sure are we it will work?
Ease (1-10): How easy is it to implement?
Score = (Impact × Confidence × Ease) / 100
Example:
Experiment: Add testimonials to homepage
Impact: 7 (could boost signups 15-20%)
Confidence: 8 (social proof is proven)
Ease: 9 (just add HTML)
ICE Score: 504 / 100 = 5.04
Sort by ICE score, run highest first
Growth Metrics Dashboard
interface GrowthMetrics {
// Acquisition
traffic_sources: {
organic: number
paid: number
referral: number
direct: number
}
cost_per_click: number
cost_per_signup: number
// Activation
signup_to_activation_rate: number
time_to_activation_p50: string // "2 days"
onboarding_completion_rate: number
// Retention
dau: number // Daily Active Users
wau: number // Weekly Active Users
mau: number // Monthly Active Users
dau_mau_ratio: number // Stickiness (should be >20%)
churn_rate_monthly: number
retention_d1: number
retention_d7: number
retention_d30: number
// Revenue
trial_to_paid_conversion: number
average_revenue_per_user: number
customer_lifetime_value: number
ltv_cac_ratio: number
// Referral
referral_invites_sent: number
viral_coefficient: number // Should be >1 for viral growth
nps: number // Net Promoter Score
// Experiments
active_experiments: number
experiments_shipped_this_month: number
win_rate: number // % experiments that improve metrics
}
Common Pitfalls
❌ Testing too many things at once: Change one variable at a time ❌ Stopping test too early: Wait for statistical significance ❌ Ignoring segments: Results vary by user type/traffic source ❌ P-hacking: Don't cherry-pick favorable metrics ❌ Small sample sizes: Need 1,000+ conversions per variant minimum ❌ Seasonal effects: Don't test during holidays/anomalies ❌ Novelty effect: Some changes work for 2 weeks then regress
Quick Start Checklist
Week 1: Foundation
- Set up analytics (Mixpanel, Amplitude, GA4)
- Define North Star Metric
- Map current funnel (AARRR)
- Identify biggest leak in funnel
- Set up A/B testing tool (Optimizely, VWO, Google Optimize)
Week 2-3: First Experiments
- Run 3 quick-win experiments
- Document results in spreadsheet
- Pick one big-bet experiment to design
- Calculate required sample sizes
Ongoing
- Run 5-10 experiments per month
- Review metrics weekly
- Document all learnings
- Focus on highest-ICE experiments
- Ship winning experiments to 100%
Summary
Great growth teams:
- ✅ Run 10+ experiments per month (high velocity)
- ✅ Focus on one North Star Metric
- ✅ Document everything (wins and losses)
- ✅ Prioritize by ICE score
- ✅ Wait for statistical significance
- ✅ Scale what works, kill what doesn't