Product Analytics Experiments

Product analytics and experiments

Event and tracking design

• One event = one user action with stable name and versioned schema.
• Required context: user_id or anonymous id, timestamp, session_id, product surface.
• Document in a tracking plan: event, properties, trigger, owner, PII classification.

Core metrics

Define denominator explicitly (eligible users, not all traffic).

Funnel analysis

1. Define steps and entry criteria.
2. Check identity stitching and time window.
3. Segment by platform, cohort, campaign (avoid Simpson's paradox surprises).

A/B test gates (before launch)

• Hypothesis and primary metric (one primary)
• Guardrail metrics (latency, errors, revenue risk)
• Randomization unit correct (user vs session)
• Sample size / MDE estimated; duration planned
• No peeking-driven early stop without sequential plan
• Feature flags and exposure logging verified in staging

Analysis

• Intent-to-treat as default; document exclusions.
• Report: point estimate, CI, practical significance (not only p-value).
• Slice only with pre-registration or clear exploration label.

Anti-patterns

• Changing primary metric after results visible.
• Multiple simultaneous experiments on same surface without interaction analysis.
• Tracking PII without legal/product review.

Output

Experiment brief: hypothesis, design, metrics, ship/kill criteria, results template.