Product Analytics

Product Analytics

Data-driven product decisions through metrics, experimentation, and behavioral analysis.

"In God we trust, all others bring data." — W. Edwards Deming

1. Metrics Framework

North Star Metric (NSM)

The single metric that best captures the core value your product delivers.

Characteristics:

• Measures customer value (not revenue)
• Reflects the core JTBD outcome
• Leading indicator of sustainable growth
• Actionable by product teams

Examples:

Anti-patterns:

• Revenue as NSM → optimizes for extraction, not value
• Downloads as NSM → optimizes for vanity, not retention
• Page views as NSM → optimizes for engagement, not outcomes

Input → NSM → Output

Input Metrics (leading, actionable)
  ├─ Acquisition: Sign-up rate, activation rate
  ├─ Engagement: Feature adoption, session frequency
  └─ Quality: NPS, support ticket rate
           ↓
    North Star Metric (health indicator)
           ↓
Output Metrics (lagging, business)
  ├─ Revenue: ARR, MRR, LTV
  ├─ Retention: Logo retention, revenue retention
  └─ Efficiency: CAC, payback period

The AARRR Framework (Pirate Metrics)

Acquisition  → Activation  → Retention  → Revenue  → Referral
   (A)           (A)          (R)         (R)         (R)

Acquisition: Users find you (SEO, ads, content, viral)
Activation: Users experience core value (first "aha" moment)
Retention: Users come back (Day-1, Day-7, Day-30)
Revenue: Users pay (conversion, expansion, upsell)
Referral: Users invite others (viral coefficient, NPS)

Key ratios:

• Activation rate: % of sign-ups who reach "aha" moment
• Retention rate: % of activated users who return
• Conversion rate: % of retained users who pay
• Viral coefficient (K): avg referrals per user; K > 1 = exponential growth

2. Funnel Analysis

Building a Funnel

Step 1: Define the critical path
  "What are the minimum steps a user must take to get value?"

Step 2: Measure conversion at each step
  Step          Users    Conversion    Drop-off
  ─────────────────────────────────────────────
  Landing       10,000   100%          —
  Sign-up        3,000   30%           70% ← Biggest drop
  Onboarding     1,500   50%           50%
  First Value      750   50%           50%
  Day-7 Return   375    50%           50%
  Paid            75     20%           80%

Analysis:

• Focus on the biggest drop-off first
• A 10% improvement at a 70% drop-off beats a 50% improvement at a 10% drop-off
• Segment funnel by channel, persona, or cohort

Funnel Segmentation

Funnel by Channel:

Channel     Landing  Sign-up  Activation  Retention
─────────────────────────────────────────────────────
Organic       5,000    1,500      750        375
Paid          4,000      800      240         72
Referral      1,000      700      490        343

Insight: Referral has highest quality (49% activation vs 16% paid)
Action: Shift budget from paid to referral program

3. Cohort Analysis

Cohort Retention Table

Cohort Sign-up    Week 0   Week 1   Week 2   Week 3   Week 4   Week 8
─────────────────────────────────────────────────────────────────────
2026-01-01        100%     45%      38%      35%      33%      30%
2026-01-08        100%     48%      41%      37%      35%      —
2026-01-15        100%     50%      43%      39%      —        —
2026-01-22        100%     52%      45%      —        —        —
2026-01-29        100%     55%      —        —        —        —

Signals:

• Flattening curve (horizontal at Week 4+): Product-Market Fit signal
• Improving cohorts (each row better than last): Product improving
• Declining cohorts: Product degrading or market shifting
• Benchmarks:
  • Consumer SaaS: 20-25% Week-4 retention = good, 30%+ = excellent
  • B2B SaaS: 40-50% Week-4 retention = good, 60%+ = excellent
  • Mobile apps: 15-20% Day-30 retention = good

Cohort Analysis by Feature

Cohort that used Feature X vs didn't:

                Week 1   Week 2   Week 4
─────────────────────────────────────────
Used Feature X    60%      55%      50%
Didn't use        35%      25%      15%
Lift              +71%    +120%    +233%

Insight: Feature X is strongly correlated with retention
Action: Prioritize onboarding users to Feature X

4. A/B Testing & Experimentation

Experiment Design

1. Hypothesis
   "We believe [change] will [impact metric] by [amount] because [reason]"

2. Success Criteria
   Primary metric: The one metric that determines go/no-go
   Secondary metrics: Guardrails (don't hurt X while improving Y)
   Guardrail metrics: Must not degrade (e.g., page load time)

3. Randomization
   - Truly random assignment (check for bias post-hoc)
   - Stable (same user always sees same variant)
   - Consistent across devices

4. Duration
   - Minimum: 1 full business cycle (7 days for weekly patterns)
   - Typical: 2 weeks
   - Maximum: Set before starting; don't run indefinitely

Sample Size Calculation

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# Baseline conversion and Minimum Detectable Effect
baseline = 0.20      # 20% current conversion
mde = 0.03           # Want to detect +3% (to 23%)
alpha = 0.05         # 5% false positive rate
power = 0.80         # 80% chance of detecting true effect

effect_size = proportion_effectsize(baseline + mde, baseline)
analysis = NormalIndPower()
sample_size = analysis.solve_power(
    effect_size=effect_size,
    alpha=alpha,
    power=power,
    ratio=1
)
# Result: ~3,500 users per variant

Quick reference:

Statistical Analysis

from scipy import stats

# Two-proportion z-test
control_visitors = 3500
control_conversions = 700      # 20%
variant_visitors = 3500
variant_conversions = 875      # 25%

z_stat, p_value = stats.proportions_ztest(
    [control_conversions, variant_conversions],
    [control_visitors, variant_visitors]
)

# Confidence interval
lift = (variant_conversions/variant_visitors - control_conversions/control_visitors)
# p < 0.05 AND practical significance → implement

Decision framework:

Early Stopping & Peeking

The Problem: Peeking at results and stopping when p < 0.05 inflates false positive rate from 5% to ~30%.

Solutions:

1. Fixed duration: Set end date before starting, don't look until then
2. Sequential testing: Use proper sequential boundaries (more complex)
3. Accept uncertainty: Some experiments need to run longer

Experiment Checklist

• Hypothesis is specific and falsifiable
• Primary metric defined before launch
• Sample size calculated and feasible
• Duration set (min 1 business cycle)
• Randomization checked for bias
• No other changes during experiment
• Results analyzed with both statistical and practical significance
• Losers are documented (prevents re-testing same idea)

5. Feature Impact Measurement

Before/After Analysis

Feature Launch Impact:

Metric          Before    After    Δ       Significant?
────────────────────────────────────────────────────────
Activation      25%       28%      +3pp    Yes (p=0.02)
Day-7 Retention  18%       19%      +1pp    No (p=0.18)
Revenue/user    $12       $14      +$2     Yes (p=0.01)

Interpretation: Feature improved activation and revenue but not retention.
Next step: Investigate why retention didn't improve.

Holdout Groups

Reserve a small % of users (5-10%) who never see the new feature:

Benefits:
- Measure long-term impact (some features have delayed effects)
- Detect negative interactions between features
- Provide clean baseline for future experiments

Trade-off: Slower to detect impact (smaller sample)

Incrementality Testing

For marketing or growth features, measure true incremental impact:

Method: Randomized controlled trial

Group A (90%): See the new growth feature
Group B (10%): Don't see it (control)

Measure: Difference in conversion between A and B

If A converts at 15% and B at 12%:
Incremental lift = 3 percentage points (25% relative)

6. Analytics Infrastructure

Event Tracking Schema

{
  "event": "purchase_completed",
  "timestamp": "2026-06-10T09:23:15Z",
  "user_id": "usr_12345",
  "session_id": "sess_67890",
  "properties": {
    "product_id": "prod_001",
    "price": 49.99,
    "currency": "USD",
    "payment_method": "credit_card",
    "coupon_applied": false
  },
  "context": {
    "page_url": "/checkout",
    "device": "mobile",
    "os": "iOS 17",
    "referrer": "email_campaign_summer"
  }
}

Required events for every product:

Tool Comparison

Dashboard Design

Executive Dashboard (weekly review):
┌─────────────────┬─────────────────┬─────────────────┐
│   NSM (WAU)     │   Revenue (MRR) │   Retention D7  │
│     45,200      │    $128,000     │      28%        │
│    ↑ 5% WoW     │    ↑ 8% WoW     │    → flat       │
└─────────────────┴─────────────────┴─────────────────┘
┌─────────────────┬─────────────────┬─────────────────┐
│  Activation     │   Conversion    │   NPS Score     │
│      32%        │      4.2%       │      42         │
│    ↑ 2pp MoM    │    ↑ 0.3pp      │    ↑ 3 pts      │
└─────────────────┴─────────────────┴─────────────────┘

Product Team Dashboard (daily):
- Funnel: Sign-up → Onboarding → Activation (last 7 days)
- Cohort: Week-1 retention by sign-up week
- Experiments: Active experiments + results
- Feature adoption: Top 10 features by usage

7. Common Analytics Pitfalls

Integration with Other Skills