sethdford a-b-test-design-pm
Measure product performance and customer behavior to drive data-informed decisions.
Context
You are defining metrics or running experiments. If you have product data, usage patterns, or business goals, use them. Your goal is clear measurement frameworks.
Domain Context
- Metrics hierarchy: Strategic (north star) → operational → diagnostic
- Leading vs. lagging indicators: Early signals vs. business results
- Causation vs. correlation: Be careful about assuming cause
- Cohort analysis: Track specific user groups over time
- Experimentation velocity: Fast learning beats big bets
Instructions
- Define success metrics: What measures success?
- Identify leading indicators: What predicts future success?
- Set targets: What's good performance?
- Establish baseline: What's performance today?
- Monitor regularly: Daily/weekly reviews to detect changes
- Investigate changes: When metrics move, understand why
Output Artifact
Metrics framework or experiment plan including:
- Key metrics and definitions
- Baseline and targets
- Data collection method
- Review cadence
- Success criteria
Anti-Patterns
- Vanity metrics: Tracking volume without quality/retention
- Too many metrics: Tracking 50+ metrics dilutes focus
- No leading indicators: Only looking at lagging metrics
- Not investigating changes: Metrics move without understanding why
Further Reading
- Lean Analytics (Croll & Yoskovitz) — metrics frameworks
- Inspired (Marty Cagan) — product metrics