# Metric Design Experimentation

> Use when designing a metric framework, selecting a North Star metric, building a metric decomposition tree, designing A/B experiments, setting up retention cohort analysis, or diagnosing whether a metric is being gamed. Encodes NSM rubrics, Goodhart's Law countermeasures, statistical validity for PMs, and retention curve methodology.

- Skill: `avyayalaya/metric-design-experimentation` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add avyayalaya/metric-design-experimentation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/avyayalaya/metric-design-experimentation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Marketing & Growth
- License: MIT
- Author: Avyayalaya (https://skillmd.com/u/avyayalaya)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/avyayalaya/metric-design-experimentation

---


## Purpose

Produce a complete Measurement Framework — metric hierarchy (North Star → L1 → L2 → input), leading/lagging indicator pairs with temporal lag classification, counter-metric design that resists Goodhart's Law, experiment plans with statistical validity, and retention cohort methodology. The output is not a dashboard mockup or a list of KPIs — it is a metric *engineering* system: instrumented to detect problems early, paired to resist gaming, and validated causally. The artifact a PM cannot produce unaided.

## When to Use / When NOT to Use

**Use this skill when:**
- Launching a new product or feature and need to define what success looks like *before* building
- Designing an A/B test or experiment plan with proper statistical rigor
- An existing metric feels "off" — you suspect proxy divergence, gaming, or Simpson's paradox
- Building a metric hierarchy for a team or org (North Star → team-level → input metrics)
- Setting up retention cohort analysis to detect PMF erosion early
- Evaluating whether a metric improvement is real or an artifact of denominator shift

**Do NOT use this skill when:**
- You need SaaS finance metric definitions (MRR, ARR, CAC, LTV formulas → use a finance metrics reference)
- You need dashboard layout or visualization design (that's a BI/design task)
- You need to analyze experiment results that already exist (use the computation scripts directly)
- You need customer research methodology (→ Discovery & Research skill — that's primary research, this is measurement design)

**Anti-inputs (what this skill does NOT handle):**
- Finance metric calculation formulas (→ SaaS finance reference skills)
- Data pipeline architecture (→ engineering)
- Dashboard UI design (→ BI tooling)
- Customer interview design (→ Discovery & Research skill)

---

## Format Rules

These rules apply to every output from this skill. They are mandatory, not optional.

### Rule 1: Take Positions with Calibrated Confidence
Never use weasel words in conclusions. Replace "likely," "may," "could," "seems" with explicit confidence levels:
- **H (>70%)** — Strong evidence (validated in your own data)
- **M (40-70%)** — Mixed or moderate evidence; direction is probable
- **L (<40%)** — Evidence is thin or conflicting; treat as hypothesis

*Why:* A high-confidence NSM selection based on validated correlation requires different action than a low-confidence hypothesis about an activation metric. Undifferentiated confidence obscures this distinction.

### Rule 2: Per-Cell Evidence Tier Annotation in Comparison Tables
Every table cell making a substantive claim carries an inline evidence tier:
- `Strong (T2)` — validated in your own experiment or cohort data
- `Adequate (T4)` — industry benchmark or comparable product data
- `Unknown (T6: inferred)` — theoretical reasoning or first principles only

*Why:* A retention target of 40% from your own validated cohorts (T2) carries different weight than a general SaaS benchmark (T4). Conflating them leads to miscalibrated goals.

### Rule 3: Metric Intervention Cascade for All Recommendations
Every metric design recommendation or triggered intervention follows:
```
OBSERVATION [evidence tier] → IMPLICATION [mechanism] → RESPONSE [specific action + owner] → CONFIDENCE [H/M/L + key assumption] → WATCH INDICATOR [observable signal]
```
*Why:* The most common failure in metric frameworks is disconnected recommendations — "increase activation rate" without specifying the mechanism, owner, or the observable signal that confirms the intervention is working.

### Rule 4: Framework Selection Before Application (Step 0)
Different metric design questions require different framework subsets. Always apply the Step 0 routing table before selecting which of the 9 frameworks to use.

*Why:* Applying all 9 frameworks to every question inflates output 2-3x without improving decision quality. An experiment design question doesn't need PMF assessment; a PMF diagnosis doesn't need MAB algorithm selection.

### Rule 5: Surface Contradictions Between Frameworks
When NSM selection criteria conflict with experiment design constraints, or retention curve analysis contradicts PMF signals, surface the contradiction explicitly. Do not resolve artificially.

*Why:* Contradictions are the most valuable signals. A product showing strong NSM growth alongside a degrading cohort retention curve has a structural problem that "reconciling" the two metrics would hide.

### Rule 6: Staleness Flags
Any benchmark, threshold, or normative value based on data older than 6 months must carry `[POTENTIALLY STALE — verify before using]`.

### Rule 7: Evidence-Limited Flags
If a metric design recommendation rests only on theoretical reasoning (T6) or general SaaS benchmarks without validation in your own data, prepend: `[EVIDENCE-LIMITED: validate with your own data before acting]`.

### Rule 8: Framework References Get One-Line Context
Not "Goodhart's Law (Regressional variant)" but "Goodhart's Law — when optimizing a proxy metric causes it to diverge from the outcome you actually care about. The 'regressional' variant means the metric was a good proxy initially but optimization pushed it past the point where it predicts the real outcome." The reader needs to understand why a framework matters for their decision, not just its academic name.

### Rule 9: The Document Must Be Navigable by Non-Creators
Include a reading guide (by time and by role), a notation key, and layered depth. A VP should be able to read only the Executive Summary and know whether the measurement plan is sound. A PM lead should be able to read through the metric hierarchy and experiment plans. A data scientist should be able to dive into statistical design. No reader should encounter unexplained notation.

---

## Output Template (Mandatory Document Skeleton)

Every Measurement Framework MUST follow this exact structure. Copy this skeleton and fill it in. Do not reorder sections, skip sections, or invent new top-level sections. If a framework was skipped in Step 0, note "Skipped — not load-bearing for this question type" in that section.

```markdown
# Measurement Framework: [Subject — e.g., "Co-Editing Feature Launch Metrics"]

> **Date:** [YYYY-MM-DD] | **Confidence band:** [Overall H/M/L] | **Staleness window:** [Date after which benchmarks and thresholds need revalidation]

---

## Executive Summary

[5 sentences max. A VP reads only this and decides whether the measurement plan is sound. No framework names, no statistical jargon, no evidence tier tags. Plain language. Final sentence = the single most important metric to watch in bold.]

---

## How to Read This Document

**What this is:** A measurement engineering system — not a KPI list. It defines what to measure, how to know if it's working, what could go wrong, and how to detect problems early.

**Reading by time available:**

| Time | Read | You'll get |
|---|---|---|
| **5 min** | Executive Summary only | Whether the measurement plan is sound + the key metric to watch |
| **15 min** | Executive Summary + Metric Hierarchy (section 2) + Experiment Plan (section 5) | What we're measuring, how we're testing, and the decision rules |
| **30 min** | Full document through Recommendations | Complete metric system with counter-metrics, retention design, and interventions |
| **Deep dive** | Everything including Appendix | Statistical design details, gaming detection, assumption stress-testing |

**Reading by role:**

| Role | Start with | Then read | Skip unless curious |
|---|---|---|---|
| VP / Exec | Executive Summary | Metric Hierarchy (section 2), Review Cadence | Statistical design, Goodhart analysis |
| PM Lead | Executive Summary | Sections 1-5 (NSM through Experiments), Recommendations | MAB algorithm selection, Statistical Validity details |
| Data Scientist / Analyst | Full document in order | Statistical Validity (section 5), Retention Cohorts (section 8), Adversarial Self-Critique | Outcome methodology (they know this) |
| Engineering Lead | Executive Summary | Instrumentation Feasibility, Experiment Plan (duration/sample/unit) | Framework theory sections |

---

## Notation Key

**Confidence levels** — applied to every metric design conclusion:
- **H (>70% confident)** — Validated in your own data. Act on it.
- **M (40-70%)** — Based on comparable products or reasonable inference. Validate before committing.
- **L (<40%)** — Hypothesis only. Must be tested before treating as a metric target.

**Evidence tiers** — how we know what we claim to know (tagged inline as T1-T6):
- **T1** — Direct internal behavioral data: your own usage analytics, experiment results (strongest)
- **T2** — Primary research on your users: well-sampled surveys, structured interviews
- **T3** — Expert analysis with methodology: published case studies, Reforge frameworks
- **T4** — Industry benchmarks: SaaS averages, Gartner norms (useful for direction, not targets)
- **T5** — Comparable product claims: competitor announcements, case studies without methodology
- **T6** — First-principles reasoning or general best practices (weakest — treat as starting hypothesis)

**Metric status indicators:**
- ✅ — Criterion met or metric healthy
- ❌ — Criterion failed or metric degrading
- ⚠️ — Warning signal; investigate

**Recommendation format** (O→I→R→C→W):
- **O**bservation — What the data shows (with evidence tier)
- **I**mplication — Why it matters (the mechanism)
- **R**esponse — What to do (specific action + owner + timeline)
- **C**onfidence — How sure we are (H/M/L + key assumption)
- **W**atch — How to know if we're wrong (observable signal)

**Flags:**
- `[POTENTIALLY STALE]` — Benchmark data is >6 months old; verify before using as a target
- `[EVIDENCE-LIMITED]` — Recommendation based on T4-T6 only; validate with your own data before acting

---

## Step 0: Context Fitness Check

Before selecting frameworks, verify that a Measurement Framework is the right artifact and that you have the data access to produce one.

| Question | If Yes | If No |
|---|---|---|
| **Do you have access to the product's usage data?** | Analysis can set validated targets (T1-T2 evidence) | All targets are benchmarks or hypotheses. Flag prominently: "Targets below are industry benchmarks (T4) — replace with your own validated thresholds before operationalizing." |
| **Has the product been live long enough for retention data?** | Retention cohort design can use real curves | Retention targets are hypothetical. Design the instrumentation to collect this data; don't set targets you can't yet validate. |
| **Is the metric system for a new product or an existing one?** | New: focus on F1 (NSM), F4 (Experiments), F9 (PMF). Existing: focus on F3 (Goodhart), F6 (Cohorts), F2 (Leading/Lagging). | — |
| **Who will operationalize this framework?** | If data team: full statistical depth. If PM without data support: simplify experiment design, focus on hierarchy + counter-metrics. | Match the framework's complexity to the team that will maintain it. A beautiful statistical design that nobody monitors is a planning artifact. |

---

## Step 0b: Framework Selection

| Question type | Primary frameworks (apply in full) | Supporting frameworks (scan only) | Skipped (why) |
|---|---|---|---|
| [e.g., "New feature launch"] | [e.g., F1 NSM + Decomposition, F2 Leading/Lagging, F3 Counter-Metrics, F4 Experiment Design] | [e.g., F8 HEART] | [e.g., "F7 MAB — insufficient traffic for multi-arm. F9 PMF — PMF already validated."] |

---

## 1. Value Moment & North Star Metric

**Value moment:** [The specific instant the user receives core value.]

**NSM Candidate Evaluation:**

| Candidate NSM | Value Reflection | Leading Nature | Influenceability | Simplicity | Non-Gameability | Score |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| [Candidate 1] | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | X/5 |
| [Candidate 2] | | | | | | X/5 |
| [Candidate 3] | | | | | | X/5 |

**Selected NSM:** [Winner + one-sentence explanation a new hire could repeat.]

**GSM validation:** Goal → [what outcome?] | Signal → [what user behavior?] | Metric → [how measured?]

---

## 2. Metric Decomposition Tree

| Level | Metric | Owner | Cadence | Target | Counter-Metric |
|---|---|---|---|---|---|
| **NSM** | [metric] | [exec] | Monthly | [target] (TX) | [counter-metric] |
| **L1** | [metric] | [PM] | Weekly | [target] (TX) | [counter-metric] |
| **L1** | [metric] | [PM] | Weekly | [target] (TX) | [counter-metric] |
| **L1** | [metric] | [PM] | Weekly | [target] (TX) | [counter-metric] |
| **L2** | [metric] | [feature team] | Daily | [target] (TX) | [counter-metric] |
| **L2** | [metric] | [feature team] | Daily | [target] (TX) | [counter-metric] |
| **Input** | [metric] | [eng lead] | Per-deploy | [target] (TX) | — |

**Causal chain check:** [Trace from each input metric to the NSM in ≤3 steps. Flag any broken branch.]

---

## 3. Leading / Lagging Indicator Pairs

| Lagging Metric | Leading Indicator | Temporal Lag | Correlation (est.) | Causal? | Alert Threshold |
|---|---|---|---|---|---|
| [metric] | [leading indicator] | Immediate/Short/Medium | r ≈ X.XX (TX) | Yes/Hypothesis | [threshold = action] |
| [metric] | [leading indicator] | | | | |

**Activation Metric (Aha Moment Protocol):**
1. **Aha moment hypothesis:** [What action predicts retention?]
2. **Correlation check:** [r value between action and retention] (TX)
3. **Threshold:** [X actions within Y days]
4. **Causal validation plan:** [Experiment to confirm causation, not just correlation]

⚠️ [Flag whether activation metric is validated or hypothesized. If hypothesized, mark as `[EVIDENCE-LIMITED]`.]

---

## 4. Counter-Metric Design & Goodhart Vulnerability

| Primary Metric | Most Likely Goodhart Variant | What Goes Wrong | Counter-Metric | Threshold | Gaming Detection Pattern |
|---|---|---|---|---|---|
| [metric] | Regressional/Extremal/Causal/Adversarial | [specific gaming scenario] | [counter-metric] | [failure threshold] | [observable signal of gaming] |
| [metric] | | | | | |
| [metric] | | | | | |

**Quarterly Health Review Protocol:**
- **Cadence:** [e.g., Every 13 weeks]
- **Owner:** [name/role]
- **Decision framework:** Keep (proxy-outcome r > 0.5) / Recalibrate (r 0.3-0.5) / Replace (r < 0.3)

---

## 5. Experiment Plan

**Experiment 1: [Title]**

| Field | Value |
|---|---|
| Hypothesis | [If we do X, metric Y will improve by ≥ Z] |
| Primary metric | [single metric] |
| Secondary metrics | [exploratory, Bonferroni-corrected at α = X] |
| Guardrail metrics | [thresholds that must hold] |
| MDE | [Xpp — business rationale for this threshold] |
| α / Power | 0.05 / 0.80 |
| Sample size | [computed via `sample_size_calculator.py` — state result or flag "script not available"] |
| Duration | [X days — rationale: covers Y cycles] |
| Randomization unit | [user/session/cluster — rationale] |
| Exclusions | [who is excluded and why] |
| Decision rule | [Ship if... Do NOT ship if...] |

**Experiment Quality Score:** X/6 (pre-registered hypothesis, single primary metric, guardrails declared, duration committed, sample size computed, segmentation planned)

**Experiment 2: [Title]**
[Same structure]

**"When NOT to Experiment" Check:**
- [ ] Is the change reversible? If not, consider staged rollout instead.
- [ ] Is there enough traffic? If sample size > 4 weeks of traffic, the experiment is infeasible.
- [ ] Is the ethical bar met? If the control group is harmed by withholding, use quasi-experimental.

---

## 6. HEART Framework (if applicable)

| Dimension | Goal | Signal | Metric | Target | Counter-Metric |
|---|---|---|---|---|---|
| Happiness | [goal] | [signal] | [metric] (TX) | [target] | [counter] |
| Engagement | | | | | |
| Adoption | | | | | |
| Retention | | | | | |
| Task Success | | | | | |

[Select 2-3 relevant dimensions. Do NOT force all five. Note "Skipped — [reason]" for unused dimensions.]

---

## 7. PMF Assessment (if applicable)

**Ellis Test:** [% who would be "very disappointed" if product disappeared] = X% (TX)
- ≥40% = PMF signal present | 25-40% = weak signal | <25% = PMF not established

**Behavioral PMF Check:**

| Signal | Value | Assessment |
|---|---|---|
| Retention curve shape | Smile / Flat / Frown | [interpretation] |
| Organic referral rate | X% (TX) | [benchmark comparison] |
| Activation-to-retention correlation | r = X.XX (TX) | [strong/weak/unvalidated] |

**Segmentation:** [Where is PMF strongest? Segment by channel, use case, activation behavior.]

---

## 8. Retention Cohort Design

**Primary cohort type:** [Time-based / Behavior-based / Channel-based]

**Retention Windows:**

| Window | Definition | Benchmark | Degradation Threshold |
|---|---|---|---|
| Day 1 | [definition] | X% (TX) | >Xpp decline vs. prior cohort |
| Day 7 | | | |
| Day 14 | | | |
| Day 30 | | | |
| Day 60 | | | |
| Day 90 | | | |

**Cohort Cuts:**
- **Time-based:** [weekly/monthly signup cohorts]
- **Behavior-based:** [activated vs. not activated — validates activation metric]
- **Channel-based:** [organic vs. paid — detects acquisition quality shifts]

**Retention Curve Shape:** [Smile (recovering) / Flat (stable) / Frown (decaying)] — interpretation and action.

**Revenue vs. Logo:** [Track both. Note if they diverge — expansion revenue can mask logo churn.]

---

## 9. Metric Intervention Recommendations (O→I→R→C→W Cascade)

**Intervention 1: [Title]**
- **Observation** [TX]: [What the data shows]
- **Implication**: [Why it matters — the mechanism]
- **Response**: [Specific action + owner + timeline]
- **Confidence**: [H/M/L] — assumes [key assumption]
- **Watch**: [Observable signal]; if [threshold], re-assess

**Intervention 2: [Title]**
- **Observation** [TX]: ...
- **Implication**: ...
- **Response**: ...
- **Confidence**: ...
- **Watch**: ...

---

## Cross-Framework Contradictions

| Contradiction | Framework A says | Framework B says | Resolution / Which to weight |
|---|---|---|---|
| [e.g., "NSM growth vs. cohort degradation"] | [NSM trending up] | [Retention cohorts degrading] | [Which matters more and why — e.g., "Cohort signal is more structural; NSM growth is masking mix shift"] |

---

## Instrumentation Feasibility

| Metric | Data exists? | Clean & reliable? | Timely (within cadence)? | ≥30 days history? | Status |
|---|:---:|:---:|:---:|:---:|---|
| [metric] | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | Ready / Needs work / Blocked |

[Flag any metric that fails ≥1 check. Propose measurable proxy or instrumentation investment.]

---

## Review Cadence & Ownership

| Metric Level | Review Cadence | Owner | Escalation Trigger |
|---|---|---|---|
| NSM | Monthly | [exec] | [e.g., ">10% decline sustained 2 months"] |
| L1 | Weekly | [PM] | [e.g., "Misses target 3 consecutive weeks"] |
| L2 | Daily | [feature team] | [e.g., "Alert threshold crossed"] |
| Experiments | Per-experiment | [PM + data] | [e.g., "Guardrail violated"] |
| Quarterly health review | Every 13 weeks | [owner] | [e.g., "Any proxy-outcome r < 0.5"] |

---

## Assumption Registry

| # | Assumption | Framework it underpins | Confidence | Evidence | What would invalidate this |
|---|---|---|---|---|---|
| 1 | | | H/M/L | (TX) | |
| 2 | | | H/M/L | (TX) | |
| 3 | | | H/M/L | (TX) | |

---

## Adversarial Self-Critique

**Weakness 1: [Title]**
[What assumption is being made? What evidence would disprove it? Is there a scenario where this metric system is catastrophically wrong? Link to a Watch Indicator.]

**Weakness 2: [Title]**
[Same depth]

**Weakness 3: [Title]**
[Same depth]

---

## Revision Triggers

| Trigger | What to re-assess | Timeline |
|---|---|---|
| NSM-outcome correlation drops below r = 0.5 | NSM selection | Next quarterly review |
| Leading indicator no longer predicts lagging | Leading/lagging pairs | Immediate investigation |
| Assumption Registry item invalidated | Dependent framework sections | Within 1 week |
| Retention curve shape changes | PMF assessment + cohort design | Next quarterly review |
| Counter-metric threshold crossed 2+ consecutive periods | Primary metric + gaming detection | Immediate investigation |

---

## Sources

[All sources cited in the framework, with evidence tier and date.]
```

**Rules for using this template:**
1. **Do not skip sections.** If a section isn't applicable (e.g., HEART skipped in Step 0), write "Skipped — [reason]" and move on.
2. **Every table cell with a target, benchmark, or claim must have an evidence tier tag** — `(T1)` through `(T6)`.
3. **Section headers are conclusions, not labels.** Replace generic headers (e.g., "Retention Analysis") with insight headers (e.g., "Day-30 Retention Is Masking a Cohort Degradation Problem") after completing the section.
4. **The Executive Summary is written last** but appears first. Do not write it until all sections are complete.
5. **Instrumentation Feasibility is not optional.** A beautiful metric that can't be measured is a planning artifact, not a measurement framework.

---

## Domain Frameworks

> This section IS the knowledge weapon. Each framework is encoded with its scoring rubrics, decision tables, and output format templates — not merely referenced. A PM using this skill designs metric systems that resist gaming, detect problems early, and validate causally; without these frameworks, the output degrades to a KPI list.

### Framework 1: North Star Framework + Metric Decomposition Tree

The foundational architecture for metric systems. A North Star Metric (NSM) captures the core value your product delivers. The decomposition tree breaks it into actionable layers.

**NSM Selection Rubric:**

Score every candidate NSM against these 5 criteria. A viable NSM scores ≥4/5:

| Criterion | Question | Score 1 (Fail) | Score 0 (Pass) |
|-----------|----------|----------------|----------------|
| **Value reflection** | Does this metric move when customers get value? | Metric can rise while customers are unhappy (e.g., pageviews from rage-clicks) | Metric movement = value delivered |
| **Leading nature** | Does this predict future outcomes, not just record past ones? | Revenue (lagging — already happened) | Weekly active usage (leads to retention and revenue) |
| **Influenceability** | Can the product team move this metric through their work? | Total market size (can't influence) | Activation rate (directly influenceable) |
| **Simplicity** | Can you explain it in one sentence to a new hire? | "Revenue-weighted DAU adjusted for seasonal variance" | "Weekly users who complete a core action" |
| **Non-gameability** | Is it hard to inflate without delivering real value? | Registered accounts (create bots) | Weekly retained users who complete value action |

**Decision point:** If your best candidate scores 3/5, strengthen it before proceeding. If multiple candidates score 5/5, pick the one closest to the value moment — the instant the user gets what they came for.

**Metric Decomposition Tree — 4 Levels:**

```
North Star Metric (NSM)
├── L1 Metrics (3-5) — Exec/PM-owned, reviewed weekly
│   ├── L2 Metrics (2-4 per L1) — Feature-team-owned, reviewed daily
│   │   └── Input Metrics — Directly manipulable by shipping code
│   ├── L2 Metrics
│   │   └── Input Metrics
│   └── L2 Metrics
│       └── Input Metrics
├── L1 Metrics
│   └── ...
└── L1 Metrics
    └── ...
```

**Decomposition Method:**
1. Start with the NSM. Ask: "What 3-5 things must ALL be true for this metric to grow?"
2. For each L1, ask: "What 2-4 sub-components drive this L1?" These are L2 metrics.
3. For each L2, ask: "What can an engineer ship this sprint to move this?" Those are input metrics.
4. **Validation check:** Can you trace from every input metric back up to the NSM with a clear causal story? If not, the tree has a broken branch.

**Metric-to-Team Ownership Mapping:**

| Level | Owner | Review Cadence | Example |
|-------|-------|---------------|---------|
| NSM | CEO / CPO | Monthly board review | Weekly active teams completing a project |
| L1 | VP / Senior PM | Weekly leadership review | Activation rate, retention rate, expansion rate |
| L2 | PM / Feature lead | Daily standup / weekly sprint | Onboarding completion, feature adoption, invite sent |
| Input | Engineering lead | Sprint-level, per-deploy | Page load time, error rate, funnel step conversion |

**GSM (Goals-Signals-Metrics) Integration:**

Before decomposing, validate the NSM using Google's GSM framework:

| Component | Question | Example |
|-----------|----------|---------|
| **Goal** | What user outcome are we trying to achieve? | Users successfully collaborate on documents in real time |
| **Signal** | What user behavior would tell us the goal is being achieved? | Multiple users editing the same document within a 5-minute window |
| **Metric** | How do we measure that signal at scale? | Weekly collaborative editing sessions (≥2 users, same doc, <5 min gap) |

**Key insight:** The signal is the hardest part. Most PMs jump from Goal to Metric and skip Signal entirely — which is how you end up measuring pageviews when you meant to measure collaboration.

**Output format — Metric Hierarchy Table:**

```
| Level | Metric | Owner | Cadence | Target | Counter-Metric |
|-------|--------|-------|---------|--------|----------------|
| NSM   | Weekly active teams completing a project | CPO | Monthly | 50K | Quality score ≥ 4.0 |
| L1    | Activation rate (day-7) | PM, Growth | Weekly | 40% | Time-to-activate ≤ 3 days |
| L1    | Week-4 retention | PM, Core | Weekly | 65% | Session depth ≥ 3 actions |
| L2    | Onboarding completion | PM, Onboarding | Daily | 70% | Support tickets from new users |
| L2    | Invite-sent rate | PM, Virality | Daily | 25% | Invite-spam complaints |
| Input | Onboarding step-3 conversion | Eng | Per-deploy | 85% | Error rate < 1% |
```

---

### Framework 2: Leading vs. Lagging Indicator Design

Every metric has a temporal character. Lagging metrics tell you what already happened. Leading metrics tell you what's about to happen. A measurement framework without leading indicators is a rearview mirror — you'll see the cliff *after* you've driven off it.

**Temporal Lag Classification:**

| Category | Lag Time | Monitoring Cadence | Examples |
|----------|----------|-------------------|----------|
| **Immediate** | Minutes to hours | Real-time alerts | Error rate, page load, API latency, crash rate |
| **Short** | 1-3 days | Daily dashboard | Daily active usage, signup rate, onboarding starts |
| **Medium** | 1-4 weeks | Weekly review | Activation rate, feature adoption, NPS responses |
| **Long** | 1-6 months | Monthly/quarterly | Retention rate, revenue, LTV, churn |
| **Structural** | 6-24 months | Quarterly strategy review | Market share, brand perception, platform adoption |

**The Lag-to-Cadence Rule:** Your monitoring cadence must match the metric's lag category. Checking a structural metric daily creates noise anxiety. Checking an immediate metric monthly means you miss fires.

**Activation Metric Design (the "Aha Moment" Protocol):**

The activation metric is the single most valuable leading indicator — it predicts long-term retention from early behavior.

| Step | Action | Example |
|------|--------|---------|
| 1 | Define the lagging outcome you want to predict | 90-day retention |
| 2 | List early user behaviors (first 7-14 days) | Created project, invited teammate, completed task, viewed report, connected integration |
| 3 | Correlate each behavior with the lagging outcome | Users who invited ≥1 teammate in week 1 → 3.2x more likely to retain at day 90 |
| 4 | Identify the top 2-3 most predictive behaviors | Invite teammate (3.2x), complete first project (2.8x), connect integration (2.1x) |
| 5 | Set threshold and timeframe | "Invited ≥1 teammate within 7 days" = activated |
| 6 | Validate: does *causing* the behavior cause retention? | Run experiment: prompt invite flow → does retention actually improve? |

**Critical warning — Step 6 is where most teams fail.** Correlation (people who invite retain better) ≠ causation (making people invite causes retention). The user who invites may simply be more engaged. You MUST run the causal experiment before declaring this your activation metric.

**Leading Indicator Discovery Protocol:**

For every lagging metric in your hierarchy, identify 2-3 leading indicators using this template:

```
| Lagging Metric | Leading Indicator | Lag Time | Correlation | Causal? | Threshold |
|---------------|-------------------|----------|-------------|---------|-----------|
| Monthly churn  | Support ticket vol | 2-3 weeks | r=0.72 | Likely (test) | >2x baseline = alert |
| Monthly churn  | Login frequency drop | 1-2 weeks | r=0.68 | Yes (validated) | <50% of avg = alert |
| Revenue growth | Pipeline velocity | 4-6 weeks | r=0.81 | Yes | <80% of target = alert |
```

---

### Framework 3: Counter-Metric Design & Goodhart's Law

> "When a measure becomes a target, it ceases to be a good measure." — Charles Goodhart

Every PM knows the quote. Almost none can name the four *mechanisms* by which it happens, or design metric systems that resist them.

**Goodhart's Law — The 4 Variants (Manheim & Garrabrant):**

| Variant | Mechanism | Example | Counter-Strategy |
|---------|-----------|---------|-----------------|
| **Regressional** | Metric was a proxy. Optimizing proxy diverges from true outcome. | Optimizing click-through rate → clickbait proliferates → user satisfaction drops | Pair with downstream quality metric (e.g., time-on-page, return rate) |
| **Extremal** | At extreme optimization, the metric-outcome relationship breaks down. | Maximizing "features shipped" → tiny features, no impact | Pair with outcome metric (feature adoption rate, not count) |
| **Causal** | Intervening on the metric destroys the causal link to the outcome. | Paying users to invite friends → invites up, quality of invitees down | Measure downstream behavior of acquired users, not just acquisition |
| **Adversarial** | Agents actively game the metric for reward. | Support team closes tickets without resolution to hit "resolution time" target | Measure reopened tickets, customer satisfaction post-resolution |

**Counter-Metric Pairing Template:**

For every primary metric, complete this row:

```
| Primary Metric | What Could Go Wrong | Counter-Metric | Threshold | Review |
|---------------|--------------------|--------------------|-----------|--------|
| Signup rate | Fake accounts, low-quality signups | Day-7 activation rate | Must stay ≥ 35% | Weekly |
| Feature adoption | Forced adoption, dark patterns | User satisfaction (CSAT) | Must stay ≥ 4.0/5 | Bi-weekly |
| Resolution time | Premature ticket closure | Reopen rate | Must stay ≤ 8% | Weekly |
| Revenue per user | Price increases erode retention | 90-day retention | Must stay ≥ 70% | Monthly |
| DAU | Notification spam drives opens | Session duration, uninstall rate | Duration ≥ 3min, uninstall ≤ 2% | Weekly |
```

**Guardrail Metric Protocol:**

Guardrails are the non-negotiables — metrics that must NOT degrade when you optimize a primary metric. Every experiment must declare guardrails *before* launch.

| Step | Action |
|------|--------|
| 1 | List what could *degrade* if your primary metric succeeds |
| 2 | Assign a guardrail metric to each risk |
| 3 | Set a threshold: "experiment fails if guardrail crosses X" |
| 4 | Pre-register guardrails in the experiment plan |
| 5 | If primary wins but guardrail fails → do NOT ship |

**Gaming Detection Patterns:**

| Signal | What It Suggests | Investigation |
|--------|-----------------|---------------|
| Sudden spike without product change | External manipulation or instrumentation error | Check by segment, channel, device |
| Metric improves only at reporting boundaries | Reporting-cycle gaming (end-of-quarter, end-of-sprint) | Plot daily, look for sawtooth pattern |
| Primary metric up, counter-metric down | Goodhart's Law is active | Activate quarterly metric health review |
| Metric improves for one segment, degrades for others | Simpson's paradox or targeted gaming | Segment decomposition, check overall vs. cohort |
| Leading and lagging indicators moving in opposite directions | Proxy divergence — leading indicator no longer predicts lagging | Re-validate correlation, consider metric rotation |

**Quarterly Metric Health Review:**

Every quarter, review each metric in the hierarchy:

| Question | If "No" → Action |
|----------|-------------------|
| Does this metric still correlate with the outcome we care about? | Re-validate. If r < 0.5 → replace or recalibrate |
| Has anyone changed behavior specifically to hit this metric? | Assess Goodhart variant. Add counter-metric or rotate |
| Is the proxy still a good proxy? | Check proxy-outcome correlation trend. Declining = divergence |
| Are counter-metrics stable? | If counter-metric degraded → primary metric gains are illusory |
| Decision | **Keep** / **Recalibrate** (adjust threshold or definition) / **Replace** (metric has been gamed out) |

---

### Framework 4: Experiment Design

Not every question needs an experiment. Not every experiment should be an A/B test. This framework encodes when to experiment, what type to use, and how to design it properly.

**Experiment Type Decision Table:**

| Question | If Yes → | If No → |
|----------|----------|---------|
| Can you randomly assign individual users? | **A/B test** (gold standard) | ↓ |
| Are there marketplace/network effects between users? | **Switchback experiment** (randomize by time window, not user) | ↓ |
| Can you find a natural comparison group? | **Quasi-experimental** (diff-in-diff, propensity matching) | ↓ |
| Do you need to stop early if results are clear? | **Sequential testing** (controls for peeking) | ↓ |
| Is traffic too low for any of the above? | **Pre/post with caution** or **qualitative validation** | Document why experiment wasn't feasible |

**A/B Test Design Protocol:**

Complete every field *before* launching. Incomplete fields = incomplete experiment.

```
| Field | Value | Notes |
|-------|-------|-------|
| Hypothesis | [If we change X, metric Y will improve by Z%] | Must be falsifiable |
| Primary metric | [ONE metric] | Pre-declared. This is what decides ship/no-ship |
| Secondary metrics | [2-3 metrics] | Exploratory. Bonferroni-corrected if used for decisions |
| Guardrail metrics | [1-3 metrics] | Must NOT degrade. Pre-set thresholds |
| MDE | [X%] | Smallest effect worth detecting (business decision, not stats) |
| Significance level (α) | [0.05 typical] | False positive tolerance |
| Power (1-β) | [0.80 minimum] | False negative tolerance |
| Sample size (N) | [computed] | Use sample_size_calculator.py |
| Duration | [X days/weeks] | Must cover full weekly cycle (min 7 days) |
| Randomization unit | [user / session / org / region] | Must match analysis unit |
| Exclusions | [who is excluded and why] | New users only? Existing only? |
| Decision rule | [Ship if primary ≥ MDE AND guardrails hold] | Written before launch |
```

**When NOT to Experiment — Checklist:**

| Condition | Why | Alternative |
|-----------|-----|------------|
| Ethical concern | Can't withhold a safety feature from control group | Ship to 100%, monitor pre/post |
| Insufficient traffic | Need 50K users but have 5K | Qualitative research, pre/post, or longer duration |
| Irreversible change | Database migration, pricing change with contracts | Staged rollout with monitoring, not true A/B |
| Legal/regulatory | GDPR, healthcare, financial regulations | Consult legal. Document decision |
| Network/spillover effects | Treatment users affect control users (social product) | Cluster randomization or switchback |
| The answer is obvious | 500ms → 50ms page load; broken flow fix | Just ship it. Experimentation has opportunity cost |

**Experiment Quality Rubric:**

Score each experiment 0-6. Below 4 = do not trust results.

| Criterion | 1 point if... |
|-----------|--------------|
| **Pre-registration** | Hypothesis, primary metric, and sample size documented before launch |
| **Single primary metric** | ONE metric decides ship/no-ship (not "whichever metric moves") |
| **Guardrails declared** | At least 1 guardrail metric with pre-set threshold |
| **Duration committed** | Full duration run (no early stopping without sequential testing framework) |
| **Sample size committed** | Computed via power analysis, not guessed |
| **Segmentation planned** | Key segments identified for heterogeneous treatment effect analysis |

---

### Framework 5: Statistical Validity for PMs

This section explains statistical concepts in plain English. No formulas. No textbook definitions. The computation scripts handle the math — this section builds the intuition to use them correctly.

**Significance (α) — "How sure are you this isn't a fluke?"**

Imagine flipping a coin 10 times and getting 7 heads. Suspicious? Maybe. Now imagine 100 flips and 70 heads. *Very* suspicious. Statistical significance measures whether your experiment result is suspicious enough to believe.

- **α = 0.05** means: "I'm accepting a 5% chance this result is a coincidence." One in twenty experiments will show a "win" that's actually noise.
- **Lower α (0.01)** = more cautious. Fewer false alarms, but you'll miss some real effects.
- **Higher α (0.10)** = more aggressive. Catch more real effects, but more false alarms too.
- **PM rule of thumb:** Use 0.05 for most tests. Use 0.01 for high-stakes, irreversible decisions (pricing, major redesigns). Use 0.10 for quick directional learnings you'll validate later.

**Power (1-β) — "Would you even notice if something real happened?"**

Power is like a metal detector's sensitivity. A cheap detector (low power) misses gold coins buried 6 inches deep. A good detector (high power) finds them.

- **Power = 0.80** means: "If there IS a real effect, I have an 80% chance of detecting it." Twenty percent of the time, a real improvement goes undetected.
- **Low power is the silent killer.** You run a test, see "no significant result," and conclude "the feature doesn't work." But maybe it DOES work — your test just wasn't powerful enough to detect it.
- **What kills power:** Small sample size, small effect, high variance in the metric.
- **PM rule of thumb:** Never run a test with power below 0.80. If your calculator says you need 100K users and you have 10K, *don't run the test* — you'll learn nothing.

**MDE (Minimum Detectable Effect) — "What's the smallest win worth knowing about?"**

Before every test, decide: "What's the smallest improvement that would change my decision?"

- If a 0.5% conversion lift isn't worth the eng effort to maintain, don't design a test to detect 0.5%. Set MDE at 2% or 5%.
- **Smaller MDE = bigger sample size = longer test.** This is a *business* tradeoff, not a statistics one.
- **The conversation to have:** "Are we willing to run this test for 6 weeks to detect a 1% lift? Or would a 3% lift be the threshold for shipping, in which case we need only 2 weeks?"

**Confidence Intervals — "Where does the truth probably live?"**

A p-value tells you "significant or not" — binary. A confidence interval tells you "the real effect is probably between X% and Y%."

- Result: "+3.2% conversion, 95% CI [0.8%, 5.6%]" means the true effect is probably between 0.8% and 5.6%. The point estimate is 3.2%, but it could be as low as 0.8%.
- **PM decision rule:** Look at the *bottom* of the confidence interval. If the worst case (0.8%) is still worth shipping, ship confidently. If the worst case crosses zero or goes negative, the result is fragile.
- **Always report CIs, not just p-values.** A "significant" result with CI [0.01%, 6.0%] is very different from [2.5%, 3.5%] — same p-value, wildly different confidence.

**Sample Size — "How long do we need to run this?"**

Sample size is determined by three inputs: your baseline rate, your MDE, and your α/power choices. It is *not* a guess.

- Use `sample_size_calculator.py` with your inputs. It outputs the required N per variant and estimated duration

…(truncated)