# Analytics Design

> Telemetry events, A/B test instrumentation, and success metrics design

- Skill: `dtsong/analytics-design` (Agent Skill)
- Install (CLI): `npx skillmds@latest add dtsong/analytics-design`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dtsong/analytics-design/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics, DevOps & Infra
- Author: dtsong (https://skillmd.com/u/dtsong)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/dtsong/analytics-design

---


# Analytics Design

## Purpose

Design a comprehensive measurement strategy including metrics hierarchy, event taxonomy, funnel instrumentation, and A/B test framework to ensure features can be evaluated with data.

## Inputs

- Feature or product area to instrument
- Business goals and key questions to answer
- Existing analytics infrastructure (tools, event patterns, naming conventions)
- Privacy requirements and data retention policies
- Target user segments for experimentation

## Process

### Step 1: Define Key Metrics Hierarchy

Build a three-tier metrics structure:
- **North Star Metric:** The single metric that best captures the value delivered to users (e.g., weekly active users completing core action)
- **Leading Indicators:** Metrics that predict movement in the North Star (e.g., activation rate, feature adoption, session frequency)
- **Guardrail Metrics:** Metrics that must NOT degrade (e.g., page load time, error rate, unsubscribe rate, support ticket volume)

For each metric, define:
- Precise definition (what counts, what doesn't)
- Data source and calculation method
- Current baseline (if known)
- Target threshold

### Step 2: Design Event Taxonomy

Establish a consistent naming convention:
- **Category:** Feature area (e.g., `editor`, `dashboard`, `onboarding`)
- **Action:** User action or system event (e.g., `clicked`, `viewed`, `completed`, `errored`)
- **Label:** Specific element or variant (e.g., `save_button`, `sidebar_nav`, `dark_mode`)
- **Value:** Numeric value if applicable (e.g., duration, count, score)

Naming rules:
- Use snake_case for all event names
- Format: `category.action.label` (e.g., `editor.clicked.save_button`)
- Keep consistent across platforms (web, mobile, API)
- Version events when schema changes (`v2.editor.clicked.save_button`)

### Step 3: Specify Event Properties

For each trackable interaction, define:

| Event Name | Properties | Type | Required | Description |
|------------|-----------|------|----------|-------------|
| `feature.action.label` | user_id | string | yes | Anonymous user identifier |
| | session_id | string | yes | Current session |
| | timestamp | ISO 8601 | yes | Event time |
| | [custom props] | ... | ... | Feature-specific data |

Include:
- Common properties (present on every event)
- Feature-specific properties
- Context properties (page, referrer, device, viewport)

### Step 4: Plan Funnel Instrumentation

For each key user flow:
- Define funnel stages (each stage = a specific event)
- Identify drop-off measurement points between stages
- Design recovery tracking (users who drop off then return)
- Plan cohort segmentation (new vs returning, plan type, entry source)

```
[Stage 1: View] → [Stage 2: Engage] → [Stage 3: Convert] → [Stage 4: Retain]
     100%              65%                  30%                  20%
         ↓ 35% drop        ↓ 35% drop          ↓ 10% drop
    [Exit survey?]    [Tooltip help?]      [Follow-up email?]
```

### Step 5: Design A/B Test Framework

For each planned experiment:
- **Experiment name:** Descriptive, following convention (e.g., `onboarding_v2_simplified_flow`)
- **Hypothesis:** "If we [change], then [metric] will [improve/decrease] because [reason]"
- **Variants:** Control (A) and treatment(s) (B, C...)
- **Allocation:** Percentage split per variant (typically 50/50 for two variants)
- **Sample size:** Calculate minimum detectable effect (MDE) with 80% power, 95% significance
- **Duration:** Estimated time to reach sample size based on traffic
- **Guardrails:** Metrics to monitor for negative effects during the test

### Step 6: Define Success Criteria

For each experiment:
- **Primary metric:** The one metric that determines success
- **Minimum detectable effect:** The smallest improvement worth shipping (e.g., +5% conversion)
- **Statistical significance threshold:** Typically p < 0.05
- **Practical significance:** Is the effect size meaningful even if statistically significant?
- **Decision framework:**
  - Significant positive result → Ship to 100%
  - Significant negative result → Revert and analyze
  - No significant result → Extend test or abandon
  - Mixed results (primary up, guardrail down) → Investigate and decide

### Step 7: Plan Data Pipeline and Storage

- **Collection method:** Client-side SDK, server-side events, or hybrid
- **Transport:** Real-time streaming vs batch upload
- **Storage:** Analytics warehouse, time-series database, or SaaS platform
- **Retention:** How long to keep raw events vs aggregated data
- **Access:** Who can query the data, dashboards to create
- **Privacy:** PII handling, consent tracking, GDPR/CCPA compliance

## Output Format

### Metrics Hierarchy

```
North Star: [Metric Name]
  ├── Leading: [Indicator 1]
  ├── Leading: [Indicator 2]
  ├── Leading: [Indicator 3]
  ├── Guardrail: [Metric A] (must not decrease)
  └── Guardrail: [Metric B] (must stay below threshold)
```

### Event Catalog

| Event Name | Description | Properties | Trigger |
|------------|-------------|-----------|---------|
| `category.action.label` | ... | `{prop1, prop2, ...}` | User clicks X |
| ... | ... | ... | ... |

### Funnel Diagram

```
[Stage 1] ──→ [Stage 2] ──→ [Stage 3] ──→ [Stage 4]
  100%           ___%           ___%           ___%
```

### A/B Test Plan

| Experiment | Hypothesis | Variants | Sample Size | Duration | Primary Metric |
|------------|-----------|----------|-------------|----------|----------------|
| ... | If..., then... | A/B | ... | ... weeks | ... |

## Quality Checks

- [ ] North star metric is clearly defined with calculation method
- [ ] Event taxonomy follows consistent naming convention
- [ ] All key user interactions have corresponding events
- [ ] Funnels have defined stages with drop-off measurement
- [ ] A/B tests have hypotheses, sample size calculations, and duration estimates
- [ ] Success criteria include both statistical and practical significance
- [ ] Privacy and data retention requirements addressed
- [ ] Guardrail metrics defined for each experiment

## Evolution Notes
<!-- Observations appended after each use -->