alexe-ev Diagnose Metric Movement
Purpose
Help teams systematically investigate why a metric moved — ruling out instrumentation errors, confounds, and alternative explanations before attributing the change to a product decision.
Skill type
Conceptual skill with calculation-aware components
Use this skill when
- A key metric has changed unexpectedly and the cause is unknown
- A team wants to attribute a metric change to a recent product change
- Multiple possible explanations need to be systematically evaluated
- A signal detected by monitoring needs root cause analysis
Do not use this skill when
- The goal is experiment result analysis with randomized assignment (use analyze-experiment-results)
- The metric change is minor and within normal variance
Required inputs
- Metric that changed (name and magnitude)
- Time period of the change
- Recent product changes or events that could be relevant
Optional inputs
- Segment breakdowns of the metric
- Instrumentation audit results
- External events (holidays, competitor moves, market changes)
- Traffic source or acquisition channel breakdown
Upstream context
Works best when:
- Baseline and normal variance are known
- Instrumentation is reliable
- Event log of product changes is available
Downstream handoff
Output can feed:
- formulate-experiment-hypothesis (diagnosed cause → testable hypothesis)
- identify-problem-opportunity (root cause → opportunity)
- detect-performance-signals (diagnosis updates signal interpretation)
Instructions
- Confirm the metric change is real: check instrumentation, data pipeline, and tracking.
- Check for data quality issues first — segment the change by source, platform, and geography.
- Identify the timing: when exactly did the change begin?
- List all product changes deployed in the relevant window.
- Check for external confounds: seasonality, competitors, market events.
- Run segment breakdowns to isolate who is driving the change.
- Rank candidate causes by likelihood and evidence.
- Recommend the most probable cause and next step to confirm.
Output
Provide:
- Metric change summary (magnitude, direction, timing)
- Data quality check results
- Instrumentation issues found (if any)
- Timeline of product changes in the window
- External confound assessment
- Segment breakdown analysis
- Ranked candidate causes
- Most probable cause with evidence
- Recommended next step to confirm
Risks / caveats
- Always check instrumentation before attributing a change to product — data issues are more common than they appear
- Correlation with a product change is not causation without a control group
- External factors (seasonality, competitor outages) are systematically underestimated