Metric Diagnosis

Metric Diagnosis Workflow

Purpose

Systematically investigate why a metric changed unexpectedly (dropped or spiked) by segmenting data, distinguishing internal vs. external factors, and testing hypotheses to identify root cause. Prevents rushing to wrong conclusions and ensures evidence-based responses.

When to Use This Workflow

Use this workflow when:

• Key metrics drop or spike unexpectedly
• Leadership asks "what happened to [metric]?"
• Post-mortem analysis needed after incident
• A/B test shows unexpected patterns
• Need to validate suspected root cause
• User behavior changed suddenly without obvious reason

Skills Sequence

This workflow applies the root-cause-diagnosis skill three times, then assesses North Star impact:

1. Root Cause Diagnosis (Phase 1)
   ↓ (Segment across 4 dimensions: People, Geography, Technology, Time)
2. Root Cause Diagnosis (Phase 2)
   ↓ (Distinguish intrinsic vs extrinsic factors)
3. Root Cause Diagnosis (Phase 3)
   ↓ (Build hypothesis table and test against data)
4. North Star Alignment
   ↓ (Assess impact on top-line metrics)
   
OUTPUT: Narrowed scope, tested hypotheses, identified root cause, recommended actions

Required Inputs

Gather this information before starting:

Metric Change Information

• The metric that changed
  • Name and definition
  • Example: "Weekly Active Users (WAU)"
• Magnitude of change
  • Absolute and percentage
  • Example: "10,000 → 9,000 (10% drop)"
• Timeframe
  • When was it noticed?
  • Over what period did change occur?
  • Example: "Noticed today, change happened over past week"

Data Access

• Segmented data availability
  • Can you segment by user type, geography, platform?
  • Are raw logs accessible?
• Comparison data
  • Historical baselines
  • Year-over-year seasonality
  • Competitor benchmarks (if available)

Recent Activity Log

• Recent releases (past 1-4 weeks)
  • Feature launches
  • Bug fixes
  • Infrastructure changes
• Running experiments
  • A/B tests active
  • Configuration changes
• External context
  • Marketing campaigns
  • News events
  • Competitor activity

Workflow Steps

Phase 0: Data Quality Verification (5 minutes)

Before investigating, confirm the data is real:

1. Check reporting systems

   • Is tracking still functional?
   • Any pipeline issues?
   • Data freshness correct?

2. Cross-reference sources

   • Do multiple data sources show same pattern?
   • Internal dashboards vs. external tools aligned?

3. Validate sample sizes

   • Sufficient data for statistical significance?
   • Time periods comparable?

Red flags that indicate data quality issues:

• Metric went to exactly zero
• Change happened at exact midnight
• Only visible in one data source
• Coincides with known tracking deployment

If data quality issue found: Fix tracking first, then re-investigate if problem persists.

If data is valid: Proceed to Phase 1.

Phase 1: Narrow the Scope (15-20 minutes)

Use the root-cause-diagnosis skill - 4 Dimension Segmentation

Ask clarifying questions across all four dimensions:

Dimension 1: People (User Segments)

Questions to ask:

• Does it affect all users equally?
• Specific demographics (age, gender, location)?
• New vs. returning users?
• Free vs. paid users?
• Power users vs. casual users?
• Specific cohorts or user types?

Request data segmentation by user attributes.

Dimension 2: Geography

Questions to ask:

• Specific countries or regions affected?
• Certain cities?
• Urban vs. rural?
• Climate zones?
• Time zones?

Request data segmentation by location.

Dimension 3: Technology (Platform)

Questions to ask:

• iOS vs. Android?
• Web vs. mobile app?
• Desktop vs. mobile web?
• Specific browser or OS versions?
• Device types?
• Network types (WiFi vs. cellular)?

Request data segmentation by platform/device.

Dimension 4: Time

Questions to ask:

• When exactly did it start?
• Gradual decline or sudden drop?
• Day of week patterns?
• Time of day patterns?
• Recurring or one-time?
• Seasonal effects?

Request time-series data with granular breakdown.

Output of Phase 1:

Narrow problem statement from:

• ❌ "Usage is down"

To:

• ✓ "iOS usage among teens in US dropped 10% starting Sept 1st"

Document:

## Narrowed Scope

**Original statement:** [Broad problem]
**Narrowed statement:** [Specific problem]

**Affected segments:**
- People: [Which user groups]
- Geography: [Which locations]
- Technology: [Which platforms]
- Time: [When and pattern]

**Unaffected segments:**
- [What stayed stable - equally important]

Phase 2: Generate Hypotheses (15-20 minutes)

Use the root-cause-diagnosis skill - Intrinsic vs. Extrinsic Factor Analysis

Brainstorm potential causes in both categories:

Intrinsic Factors (Internal Changes)

Engineering & Product:

• Recent code releases (check deploy logs)
• Bug introductions
• Feature launches by any team
• A/B experiments started/stopped
• Algorithm or ranking changes
• Performance degradations

Check with:

• Engineering team (deploys, errors, performance)
• Other product teams (their launches)
• Data team (tracking changes)

Operations & Business:

• Pricing changes
• Policy updates
• Support process changes
• Marketing campaign changes

Check with:

• Operations team
• Marketing team
• Customer success team

Extrinsic Factors (External Events)

Competition:

• Competitor product launches
• Competitor pricing changes
• Competitor marketing campaigns

Market:

• Economic shifts
• Industry trends
• Regulatory changes
• Platform policy changes (iOS, Android, etc.)

Calendar/Seasonal:

• Holidays
• School schedules
• Weather patterns
• Cultural events

News/PR:

• News coverage (positive or negative)
• Social media trends
• External events affecting user behavior

Check with:

• Sales team (competitive intelligence)
• Customer success (user feedback)
• Marketing (market trends)
• Industry news sources

Output of Phase 2:

List of 4-6 hypotheses (mix of intrinsic and extrinsic):

## Hypothesis List

**Intrinsic (Internal):**
1. [Hypothesis name]: [Description]
2. [Hypothesis name]: [Description]
3. [Hypothesis name]: [Description]

**Extrinsic (External):**
4. [Hypothesis name]: [Description]
5. [Hypothesis name]: [Description]
6. [Hypothesis name]: [Description]

Phase 3: Test Hypotheses (20-30 minutes)

Use the root-cause-diagnosis skill - Hypothesis Table Method

Step 3.1: Create Hypothesis Table

Columns: Data points observed (from Phase 1) Rows: Potential causes (from Phase 2)

Fill each cell with predicted impact:

• ↓ = Would decrease this segment
• ↑ = Would increase this segment
• → = No effect on this segment
• ? = Uncertain

Step 3.2: Match Patterns

For each hypothesis:

• Do predictions match observations?
• If ALL predictions match → Possible cause
• If ANY predictions contradict → Rule out

Step 3.3: Request Differentiating Data

If multiple hypotheses fit:

• Identify data points that would differ between them
• Request additional segmentation
• Narrow to single most likely cause

Example Table:

Pattern match: "School started (US)" fits all observations → Most likely cause

Output of Phase 3:

## Hypothesis Testing Results

**Ruled out:**
- [Hypothesis]: [Why ruled out]
- [Hypothesis]: [Why ruled out]

**Possible causes (remaining):**
1. [Most likely hypothesis]
   - Evidence supporting: [List]
   - Evidence against: [List if any]
   - Confidence: [High/Medium/Low]

2. [Second most likely] (if applicable)
   - Evidence supporting: [List]
   - Confidence: [High/Medium/Low]

**Additional data needed to confirm:**
- [Data point that would differentiate remaining hypotheses]

Phase 4: Assess North Star Impact (10 minutes)

Use the north-star-alignment skill

Evaluate whether identified root cause impacts company-level metrics:

Questions:

• Does this affect North Star metrics?
• Is impact temporary or permanent?
• Does it require immediate action?
• What's the business impact (revenue, growth, retention)?

Severity classification:

Critical (Act immediately):

• Impacts revenue directly
• Affects North Star metrics substantially
• Permanent degradation if not fixed
• Competitive disadvantage

Important (Act soon):

• Impacts user experience
• Affects intermediate metrics
• May worsen over time
• Needs monitoring

Low Priority (Monitor):

• Expected variation (seasonality)
• Affects non-critical segments
• Self-resolving
• No action needed

Output of Phase 4:

## North Star Impact Assessment

**Affected North Star metrics:**
- [Metric]: [Impact description]
- [Metric]: [Impact description]

**Business impact:**
- Revenue: [Direct/Indirect/None]
- Growth: [Slowed/Accelerated/Neutral]
- Retention: [Harmed/Improved/Neutral]

**Severity:** [Critical/Important/Low Priority]

**Urgency:** [Immediate/This week/Monitor]

Phase 5: Recommend Actions (10 minutes)

Based on root cause and severity, recommend next steps:

Action categories:

1. Fix (for bugs/issues)

   • What needs to be fixed?
   • Who owns the fix?
   • Timeline for fix?

2. Mitigate (for external factors)

   • What can reduce harm?
   • Counter-measures available?
   • Timeline for mitigation?

3. Monitor (for expected variations)

   • What tracking is needed?
   • Alert thresholds?
   • Review cadence?

4. No Action (for acceptable changes)

   • Why no action needed?
   • What validates this decision?

Output document:

# Metric Diagnosis Report: [Metric Name]

## Executive Summary
[One paragraph: What happened, why, what to do]

## Problem Statement
- **Initial observation:** [Broad statement]
- **Narrowed scope:** [Specific statement with segments]
- **Magnitude:** [Absolute and % change]
- **Timeframe:** [When occurred]

## Root Cause Analysis

### Narrowed Scope (Phase 1)
- **Affected:** [User segments, geography, platforms, time pattern]
- **Unaffected:** [What stayed stable]

### Hypotheses Tested (Phase 2-3)
| Hypothesis | Category | Evidence | Conclusion |
|------------|----------|----------|------------|
| [Name] | Intrinsic/Extrinsic | [Supporting/Contradicting] | Ruled Out/Possible/Likely |

### Identified Root Cause
**Most likely cause:** [Hypothesis name]

**Evidence supporting:**
1. [Data point matching prediction]
2. [Data point matching prediction]
3. [Stakeholder confirmation]

**Confidence level:** [High/Medium/Low]

### North Star Impact (Phase 4)
- **Affected metrics:** [List with impact]
- **Severity:** [Critical/Important/Low]
- **Urgency:** [Immediate/Soon/Monitor]

## Recommended Actions

### Primary action:
**[Fix/Mitigate/Monitor/No Action]**
- What: [Specific action]
- Who: [Owner]
- When: [Timeline]
- Success criteria: [How to know it worked]

### Secondary actions:
- [Action 1]
- [Action 2]

### Monitoring plan:
- Metrics to track: [List]
- Review frequency: [Daily/Weekly]
- Alert thresholds: [When to escalate]

## Appendix
- Hypothesis table: [Detailed]
- Data sources: [Links]
- Stakeholders consulted: [Names/teams]

Common Diagnostic Patterns

Pattern 1: Seasonal Variation

• Symptoms: Recurring pattern (weekly, yearly)
• Investigation: Compare to historical same period
• Action: Usually monitor, adjust baselines
• Example: School year start, holiday season

Pattern 2: Competitor Impact

• Symptoms: Sudden shift, specific demographic
• Investigation: Check competitor news, market research
• Action: Counter-strategy, feature parity
• Example: Competing app launch with targeted marketing

Pattern 3: Internal Bug

• Symptoms: Sudden drop, platform-specific
• Investigation: Check recent releases, error logs
• Action: Hotfix, rollback if needed
• Example: iOS update broke key feature

Pattern 4: Product Change Side Effect

• Symptoms: Gradual shift after release
• Investigation: Analyze feature adoption patterns
• Action: Iterate on feature, A/B test variations
• Example: Redesign changed user behavior

Pattern 5: Tracking Issue (False Alarm)

• Symptoms: Exact numbers, midnight timing, single source
• Investigation: Verify data pipeline
• Action: Fix tracking, dismiss false alarm
• Example: Analytics tag removed in deploy

Common Mistakes

Success Criteria

Metric diagnosis succeeds when:

• Data quality verified
• Problem narrowed to specific segments (4 dimensions)
• 4-6 hypotheses brainstormed (intrinsic + extrinsic)
• Hypotheses tested against data systematically
• Root cause identified (or top 2-3 candidates with confidence levels)
• North Star impact assessed
• Recommended actions clear with owners and timelines
• Diagnosis document created and shared
• Stakeholders aligned on next steps

Real-World Example: Microsoft Teams Downloads Drop

Initial Report

"New app downloads down 50% from last week"

Phase 1: Narrow Scope (15 min)

Questions asked:

• Geography? → "Predominantly US and Europe"
• User type? → "Enterprise customers, not consumers"
• Platform? → "Web and desktop primarily, some mobile"
• Other metrics? → "Usage unchanged (existing users still active)"

Narrowed statement: "Enterprise customer downloads via web/desktop down 50% in US/Europe, starting last week. Existing user engagement unchanged."

Phase 2: Generate Hypotheses (15 min)

Intrinsic:

1. Bug in download flow
2. Recent release broke something
3. A/B test reducing visibility

Extrinsic: 4. Marketing promotion ended 5. Competitor launch 6. Economic downturn affecting enterprise spending

Stakeholder consults:

• Engineering: No recent releases, no bugs reported
• Marketing: "We were running 50% off for enterprises switching from Slack"
• Sales: No competitor news, pipeline healthy

Phase 3: Test Hypotheses (20 min)

Pattern match: "Promotion ended" fits all observations

Confirmation: Marketing confirms promotion ended last weekend

Phase 4: North Star Impact (5 min)

Root cause: End of promotional campaign

• Downloads during promo = artificially inflated baseline
• Current downloads = return to normal baseline

North Star impact:

• MAU: No impact (existing users unaffected)
• Revenue: No impact (existing customers retained)
• Severity: Low (false alarm, not actual problem)

Phase 5: Recommended Actions (5 min)

Primary action: No action needed

• Not an actual decline, just return to baseline
• Adjust baselines for future comparisons

Secondary actions:

• Update dashboards to flag promotional periods
• Establish "normal" baseline for future monitoring
• Document for future reference (prevent false alarms)

Time to diagnosis: 60 minutes

Related Skills

This workflow orchestrates these skills:

• root-cause-diagnosis (Phases 1-3) - Core investigation method
• north-star-alignment (Phase 4) - Assess strategic impact

Related Workflows

• tradeoff-decision: May use diagnosis to understand why metrics changed before deciding
• metrics-definition: Uses similar systematic approach for defining vs. diagnosing

Time Estimate

Total: 60-90 minutes

• Phase 0 (Data quality): 5 min
• Phase 1 (Narrow scope): 15-20 min
• Phase 2 (Hypotheses): 15-20 min
• Phase 3 (Test): 20-30 min
• Phase 4 (Impact): 10 min
• Phase 5 (Actions): 10 min

For complex issues, may require multiple iterations or longer investigation time.