Analytics Design

Analytics Design

Purpose

Design a comprehensive measurement strategy including metrics hierarchy, event taxonomy, funnel instrumentation, and A/B test framework to ensure features can be evaluated with data.

Inputs

• Feature or product area to instrument
• Business goals and key questions to answer
• Existing analytics infrastructure (tools, event patterns, naming conventions)
• Privacy requirements and data retention policies
• Target user segments for experimentation

Process

Step 1: Define Key Metrics Hierarchy

Build a three-tier metrics structure:

• North Star Metric: The single metric that best captures the value delivered to users (e.g., weekly active users completing core action)
• Leading Indicators: Metrics that predict movement in the North Star (e.g., activation rate, feature adoption, session frequency)
• Guardrail Metrics: Metrics that must NOT degrade (e.g., page load time, error rate, unsubscribe rate, support ticket volume)

For each metric, define:

• Precise definition (what counts, what doesn't)
• Data source and calculation method
• Current baseline (if known)
• Target threshold

Step 2: Design Event Taxonomy

Establish a consistent naming convention:

• Category: Feature area (e.g., editor, dashboard, onboarding)
• Action: User action or system event (e.g., clicked, viewed, completed, errored)
• Label: Specific element or variant (e.g., save_button, sidebar_nav, dark_mode)
• Value: Numeric value if applicable (e.g., duration, count, score)

Naming rules:

• Use snake_case for all event names
• Format: category.action.label (e.g., editor.clicked.save_button)
• Keep consistent across platforms (web, mobile, API)
• Version events when schema changes (v2.editor.clicked.save_button)

Step 3: Specify Event Properties

For each trackable interaction, define:

Include:

• Common properties (present on every event)
• Feature-specific properties
• Context properties (page, referrer, device, viewport)

Step 4: Plan Funnel Instrumentation

For each key user flow:

• Define funnel stages (each stage = a specific event)
• Identify drop-off measurement points between stages
• Design recovery tracking (users who drop off then return)
• Plan cohort segmentation (new vs returning, plan type, entry source)

[Stage 1: View] → [Stage 2: Engage] → [Stage 3: Convert] → [Stage 4: Retain]
     100%              65%                  30%                  20%
         ↓ 35% drop        ↓ 35% drop          ↓ 10% drop
    [Exit survey?]    [Tooltip help?]      [Follow-up email?]

Step 5: Design A/B Test Framework

For each planned experiment:

• Experiment name: Descriptive, following convention (e.g., onboarding_v2_simplified_flow)
• Hypothesis: "If we [change], then [metric] will [improve/decrease] because [reason]"
• Variants: Control (A) and treatment(s) (B, C...)
• Allocation: Percentage split per variant (typically 50/50 for two variants)
• Sample size: Calculate minimum detectable effect (MDE) with 80% power, 95% significance
• Duration: Estimated time to reach sample size based on traffic
• Guardrails: Metrics to monitor for negative effects during the test

Step 6: Define Success Criteria

For each experiment:

• Primary metric: The one metric that determines success
• Minimum detectable effect: The smallest improvement worth shipping (e.g., +5% conversion)
• Statistical significance threshold: Typically p < 0.05
• Practical significance: Is the effect size meaningful even if statistically significant?
• Decision framework:
  • Significant positive result → Ship to 100%
  • Significant negative result → Revert and analyze
  • No significant result → Extend test or abandon
  • Mixed results (primary up, guardrail down) → Investigate and decide

Step 7: Plan Data Pipeline and Storage

• Collection method: Client-side SDK, server-side events, or hybrid
• Transport: Real-time streaming vs batch upload
• Storage: Analytics warehouse, time-series database, or SaaS platform
• Retention: How long to keep raw events vs aggregated data
• Access: Who can query the data, dashboards to create
• Privacy: PII handling, consent tracking, GDPR/CCPA compliance

Output Format

Metrics Hierarchy

North Star: [Metric Name]
  ├── Leading: [Indicator 1]
  ├── Leading: [Indicator 2]
  ├── Leading: [Indicator 3]
  ├── Guardrail: [Metric A] (must not decrease)
  └── Guardrail: [Metric B] (must stay below threshold)

Event Catalog

Funnel Diagram

[Stage 1] ──→ [Stage 2] ──→ [Stage 3] ──→ [Stage 4]
  100%           ___%           ___%           ___%

A/B Test Plan

Quality Checks

• North star metric is clearly defined with calculation method
• Event taxonomy follows consistent naming convention
• All key user interactions have corresponding events
• Funnels have defined stages with drop-off measurement
• A/B tests have hypotheses, sample size calculations, and duration estimates
• Success criteria include both statistical and practical significance
• Privacy and data retention requirements addressed
• Guardrail metrics defined for each experiment

Evolution Notes