Metric Design Experimentation

Purpose

Produce a complete Measurement Framework — metric hierarchy (North Star → L1 → L2 → input), leading/lagging indicator pairs with temporal lag classification, counter-metric design that resists Goodhart's Law, experiment plans with statistical validity, and retention cohort methodology. The output is not a dashboard mockup or a list of KPIs — it is a metric engineering system: instrumented to detect problems early, paired to resist gaming, and validated causally. The artifact a PM cannot produce unaided.

When to Use / When NOT to Use

Use this skill when:

• Launching a new product or feature and need to define what success looks like before building
• Designing an A/B test or experiment plan with proper statistical rigor
• An existing metric feels "off" — you suspect proxy divergence, gaming, or Simpson's paradox
• Building a metric hierarchy for a team or org (North Star → team-level → input metrics)
• Setting up retention cohort analysis to detect PMF erosion early
• Evaluating whether a metric improvement is real or an artifact of denominator shift

Do NOT use this skill when:

• You need SaaS finance metric definitions (MRR, ARR, CAC, LTV formulas → use a finance metrics reference)
• You need dashboard layout or visualization design (that's a BI/design task)
• You need to analyze experiment results that already exist (use the computation scripts directly)
• You need customer research methodology (→ Discovery & Research skill — that's primary research, this is measurement design)

Anti-inputs (what this skill does NOT handle):

• Finance metric calculation formulas (→ SaaS finance reference skills)
• Data pipeline architecture (→ engineering)
• Dashboard UI design (→ BI tooling)
• Customer interview design (→ Discovery & Research skill)

Format Rules

These rules apply to every output from this skill. They are mandatory, not optional.

Rule 1: Take Positions with Calibrated Confidence

Never use weasel words in conclusions. Replace "likely," "may," "could," "seems" with explicit confidence levels:

• H (>70%) — Strong evidence (validated in your own data)
• M (40-70%) — Mixed or moderate evidence; direction is probable
• L (<40%) — Evidence is thin or conflicting; treat as hypothesis

Why: A high-confidence NSM selection based on validated correlation requires different action than a low-confidence hypothesis about an activation metric. Undifferentiated confidence obscures this distinction.

Rule 2: Per-Cell Evidence Tier Annotation in Comparison Tables

Every table cell making a substantive claim carries an inline evidence tier:

• Strong (T2) — validated in your own experiment or cohort data
• Adequate (T4) — industry benchmark or comparable product data
• Unknown (T6: inferred) — theoretical reasoning or first principles only

Why: A retention target of 40% from your own validated cohorts (T2) carries different weight than a general SaaS benchmark (T4). Conflating them leads to miscalibrated goals.

Rule 3: Metric Intervention Cascade for All Recommendations

Every metric design recommendation or triggered intervention follows:

OBSERVATION [evidence tier] → IMPLICATION [mechanism] → RESPONSE [specific action + owner] → CONFIDENCE [H/M/L + key assumption] → WATCH INDICATOR [observable signal]

Why: The most common failure in metric frameworks is disconnected recommendations — "increase activation rate" without specifying the mechanism, owner, or the observable signal that confirms the intervention is working.

Rule 4: Framework Selection Before Application (Step 0)

Different metric design questions require different framework subsets. Always apply the Step 0 routing table before selecting which of the 9 frameworks to use.

Why: Applying all 9 frameworks to every question inflates output 2-3x without improving decision quality. An experiment design question doesn't need PMF assessment; a PMF diagnosis doesn't need MAB algorithm selection.

Rule 5: Surface Contradictions Between Frameworks

When NSM selection criteria conflict with experiment design constraints, or retention curve analysis contradicts PMF signals, surface the contradiction explicitly. Do not resolve artificially.

Why: Contradictions are the most valuable signals. A product showing strong NSM growth alongside a degrading cohort retention curve has a structural problem that "reconciling" the two metrics would hide.

Rule 6: Staleness Flags

Any benchmark, threshold, or normative value based on data older than 6 months must carry [POTENTIALLY STALE — verify before using].

Rule 7: Evidence-Limited Flags

If a metric design recommendation rests only on theoretical reasoning (T6) or general SaaS benchmarks without validation in your own data, prepend: [EVIDENCE-LIMITED: validate with your own data before acting].

Rule 8: Framework References Get One-Line Context

Not "Goodhart's Law (Regressional variant)" but "Goodhart's Law — when optimizing a proxy metric causes it to diverge from the outcome you actually care about. The 'regressional' variant means the metric was a good proxy initially but optimization pushed it past the point where it predicts the real outcome." The reader needs to understand why a framework matters for their decision, not just its academic name.

Rule 9: The Document Must Be Navigable by Non-Creators

Include a reading guide (by time and by role), a notation key, and layered depth. A VP should be able to read only the Executive Summary and know whether the measurement plan is sound. A PM lead should be able to read through the metric hierarchy and experiment plans. A data scientist should be able to dive into statistical design. No reader should encounter unexplained notation.

Output Template (Mandatory Document Skeleton)

Every Measurement Framework MUST follow this exact structure. Copy this skeleton and fill it in. Do not reorder sections, skip sections, or invent new top-level sections. If a framework was skipped in Step 0, note "Skipped — not load-bearing for this question type" in that section.

# Measurement Framework: [Subject — e.g., "Co-Editing Feature Launch Metrics"]

> **Date:** [YYYY-MM-DD] | **Confidence band:** [Overall H/M/L] | **Staleness window:** [Date after which benchmarks and thresholds need revalidation]

---

## Executive Summary

[5 sentences max. A VP reads only this and decides whether the measurement plan is sound. No framework names, no statistical jargon, no evidence tier tags. Plain language. Final sentence = the single most important metric to watch in bold.]

---

## How to Read This Document

**What this is:** A measurement engineering system — not a KPI list. It defines what to measure, how to know if it's working, what could go wrong, and how to detect problems early.

**Reading by time available:**

| Time | Read | You'll get |
|---|---|---|
| **5 min** | Executive Summary only | Whether the measurement plan is sound + the key metric to watch |
| **15 min** | Executive Summary + Metric Hierarchy (section 2) + Experiment Plan (section 5) | What we're measuring, how we're testing, and the decision rules |
| **30 min** | Full document through Recommendations | Complete metric system with counter-metrics, retention design, and interventions |
| **Deep dive** | Everything including Appendix | Statistical design details, gaming detection, assumption stress-testing |

**Reading by role:**

| Role | Start with | Then read | Skip unless curious |
|---|---|---|---|
| VP / Exec | Executive Summary | Metric Hierarchy (section 2), Review Cadence | Statistical design, Goodhart analysis |
| PM Lead | Executive Summary | Sections 1-5 (NSM through Experiments), Recommendations | MAB algorithm selection, Statistical Validity details |
| Data Scientist / Analyst | Full document in order | Statistical Validity (section 5), Retention Cohorts (section 8), Adversarial Self-Critique | Outcome methodology (they know this) |
| Engineering Lead | Executive Summary | Instrumentation Feasibility, Experiment Plan (duration/sample/unit) | Framework theory sections |

---

## Notation Key

**Confidence levels** — applied to every metric design conclusion:
- **H (>70% confident)** — Validated in your own data. Act on it.
- **M (40-70%)** — Based on comparable products or reasonable inference. Validate before committing.
- **L (<40%)** — Hypothesis only. Must be tested before treating as a metric target.

**Evidence tiers** — how we know what we claim to know (tagged inline as T1-T6):
- **T1** — Direct internal behavioral data: your own usage analytics, experiment results (strongest)
- **T2** — Primary research on your users: well-sampled surveys, structured interviews
- **T3** — Expert analysis with methodology: published case studies, Reforge frameworks
- **T4** — Industry benchmarks: SaaS averages, Gartner norms (useful for direction, not targets)
- **T5** — Comparable product claims: competitor announcements, case studies without methodology
- **T6** — First-principles reasoning or general best practices (weakest — treat as starting hypothesis)

**Metric status indicators:**
- ✅ — Criterion met or metric healthy
- ❌ — Criterion failed or metric degrading
- ⚠️ — Warning signal; investigate

**Recommendation format** (O→I→R→C→W):
- **O**bservation — What the data shows (with evidence tier)
- **I**mplication — Why it matters (the mechanism)
- **R**esponse — What to do (specific action + owner + timeline)
- **C**onfidence — How sure we are (H/M/L + key assumption)
- **W**atch — How to know if we're wrong (observable signal)

**Flags:**
- `[POTENTIALLY STALE]` — Benchmark data is >6 months old; verify before using as a target
- `[EVIDENCE-LIMITED]` — Recommendation based on T4-T6 only; validate with your own data before acting

---

## Step 0: Context Fitness Check

Before selecting frameworks, verify that a Measurement Framework is the right artifact and that you have the data access to produce one.

| Question | If Yes | If No |
|---|---|---|
| **Do you have access to the product's usage data?** | Analysis can set validated targets (T1-T2 evidence) | All targets are benchmarks or hypotheses. Flag prominently: "Targets below are industry benchmarks (T4) — replace with your own validated thresholds before operationalizing." |
| **Has the product been live long enough for retention data?** | Retention cohort design can use real curves | Retention targets are hypothetical. Design the instrumentation to collect this data; don't set targets you can't yet validate. |
| **Is the metric system for a new product or an existing one?** | New: focus on F1 (NSM), F4 (Experiments), F9 (PMF). Existing: focus on F3 (Goodhart), F6 (Cohorts), F2 (Leading/Lagging). | — |
| **Who will operationalize this framework?** | If data team: full statistical depth. If PM without data support: simplify experiment design, focus on hierarchy + counter-metrics. | Match the framework's complexity to the team that will maintain it. A beautiful statistical design that nobody monitors is a planning artifact. |

---

## Step 0b: Framework Selection

| Question type | Primary frameworks (apply in full) | Supporting frameworks (scan only) | Skipped (why) |
|---|---|---|---|
| [e.g., "New feature launch"] | [e.g., F1 NSM + Decomposition, F2 Leading/Lagging, F3 Counter-Metrics, F4 Experiment Design] | [e.g., F8 HEART] | [e.g., "F7 MAB — insufficient traffic for multi-arm. F9 PMF — PMF already validated."] |

---

## 1. Value Moment & North Star Metric

**Value moment:** [The specific instant the user receives core value.]

**NSM Candidate Evaluation:**

| Candidate NSM | Value Reflection | Leading Nature | Influenceability | Simplicity | Non-Gameability | Score |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| [Candidate 1] | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | X/5 |
| [Candidate 2] | | | | | | X/5 |
| [Candidate 3] | | | | | | X/5 |

**Selected NSM:** [Winner + one-sentence explanation a new hire could repeat.]

**GSM validation:** Goal → [what outcome?] | Signal → [what user behavior?] | Metric → [how measured?]

---

## 2. Metric Decomposition Tree

| Level | Metric | Owner | Cadence | Target | Counter-Metric |
|---|---|---|---|---|---|
| **NSM** | [metric] | [exec] | Monthly | [target] (TX) | [counter-metric] |
| **L1** | [metric] | [PM] | Weekly | [target] (TX) | [counter-metric] |
| **L1** | [metric] | [PM] | Weekly | [target] (TX) | [counter-metric] |
| **L1** | [metric] | [PM] | Weekly | [target] (TX) | [counter-metric] |
| **L2** | [metric] | [feature team] | Daily | [target] (TX) | [counter-metric] |
| **L2** | [metric] | [feature team] | Daily | [target] (TX) | [counter-metric] |
| **Input** | [metric] | [eng lead] | Per-deploy | [target] (TX) | — |

**Causal chain check:** [Trace from each input metric to the NSM in ≤3 steps. Flag any broken branch.]

---

## 3. Leading / Lagging Indicator Pairs

| Lagging Metric | Leading Indicator | Temporal Lag | Correlation (est.) | Causal? | Alert Threshold |
|---|---|---|---|---|---|
| [metric] | [leading indicator] | Immediate/Short/Medium | r ≈ X.XX (TX) | Yes/Hypothesis | [threshold = action] |
| [metric] | [leading indicator] | | | | |

**Activation Metric (Aha Moment Protocol):**
1. **Aha moment hypothesis:** [What action predicts retention?]
2. **Correlation check:** [r value between action and retention] (TX)
3. **Threshold:** [X actions within Y days]
4. **Causal validation plan:** [Experiment to confirm causation, not just correlation]

⚠️ [Flag whether activation metric is validated or hypothesized. If hypothesized, mark as `[EVIDENCE-LIMITED]`.]

---

## 4. Counter-Metric Design & Goodhart Vulnerability

| Primary Metric | Most Likely Goodhart Variant | What Goes Wrong | Counter-Metric | Threshold | Gaming Detection Pattern |
|---|---|---|---|---|---|
| [metric] | Regressional/Extremal/Causal/Adversarial | [specific gaming scenario] | [counter-metric] | [failure threshold] | [observable signal of gaming] |
| [metric] | | | | | |
| [metric] | | | | | |

**Quarterly Health Review Protocol:**
- **Cadence:** [e.g., Every 13 weeks]
- **Owner:** [name/role]
- **Decision framework:** Keep (proxy-outcome r > 0.5) / Recalibrate (r 0.3-0.5) / Replace (r < 0.3)

---

## 5. Experiment Plan

**Experiment 1: [Title]**

| Field | Value |
|---|---|
| Hypothesis | [If we do X, metric Y will improve by ≥ Z] |
| Primary metric | [single metric] |
| Secondary metrics | [exploratory, Bonferroni-corrected at α = X] |
| Guardrail metrics | [thresholds that must hold] |
| MDE | [Xpp — business rationale for this threshold] |
| α / Power | 0.05 / 0.80 |
| Sample size | [computed via `sample_size_calculator.py` — state result or flag "script not available"] |
| Duration | [X days — rationale: covers Y cycles] |
| Randomization unit | [user/session/cluster — rationale] |
| Exclusions | [who is excluded and why] |
| Decision rule | [Ship if... Do NOT ship if...] |

**Experiment Quality Score:** X/6 (pre-registered hypothesis, single primary metric, guardrails declared, duration committed, sample size computed, segmentation planned)

**Experiment 2: [Title]**
[Same structure]

**"When NOT to Experiment" Check:**
- [ ] Is the change reversible? If not, consider staged rollout instead.
- [ ] Is there enough traffic? If sample size > 4 weeks of traffic, the experiment is infeasible.
- [ ] Is the ethical bar met? If the control group is harmed by withholding, use quasi-experimental.

---

## 6. HEART Framework (if applicable)

| Dimension | Goal | Signal | Metric | Target | Counter-Metric |
|---|---|---|---|---|---|
| Happiness | [goal] | [signal] | [metric] (TX) | [target] | [counter] |
| Engagement | | | | | |
| Adoption | | | | | |
| Retention | | | | | |
| Task Success | | | | | |

[Select 2-3 relevant dimensions. Do NOT force all five. Note "Skipped — [reason]" for unused dimensions.]

---

## 7. PMF Assessment (if applicable)

**Ellis Test:** [% who would be "very disappointed" if product disappeared] = X% (TX)
- ≥40% = PMF signal present | 25-40% = weak signal | <25% = PMF not established

**Behavioral PMF Check:**

| Signal | Value | Assessment |
|---|---|---|
| Retention curve shape | Smile / Flat / Frown | [interpretation] |
| Organic referral rate | X% (TX) | [benchmark comparison] |
| Activation-to-retention correlation | r = X.XX (TX) | [strong/weak/unvalidated] |

**Segmentation:** [Where is PMF strongest? Segment by channel, use case, activation behavior.]

---

## 8. Retention Cohort Design

**Primary cohort type:** [Time-based / Behavior-based / Channel-based]

**Retention Windows:**

| Window | Definition | Benchmark | Degradation Threshold |
|---|---|---|---|
| Day 1 | [definition] | X% (TX) | >Xpp decline vs. prior cohort |
| Day 7 | | | |
| Day 14 | | | |
| Day 30 | | | |
| Day 60 | | | |
| Day 90 | | | |

**Cohort Cuts:**
- **Time-based:** [weekly/monthly signup cohorts]
- **Behavior-based:** [activated vs. not activated — validates activation metric]
- **Channel-based:** [organic vs. paid — detects acquisition quality shifts]

**Retention Curve Shape:** [Smile (recovering) / Flat (stable) / Frown (decaying)] — interpretation and action.

**Revenue vs. Logo:** [Track both. Note if they diverge — expansion revenue can mask logo churn.]

---

## 9. Metric Intervention Recommendations (O→I→R→C→W Cascade)

**Intervention 1: [Title]**
- **Observation** [TX]: [What the data shows]
- **Implication**: [Why it matters — the mechanism]
- **Response**: [Specific action + owner + timeline]
- **Confidence**: [H/M/L] — assumes [key assumption]
- **Watch**: [Observable signal]; if [threshold], re-assess

**Intervention 2: [Title]**
- **Observation** [TX]: ...
- **Implication**: ...
- **Response**: ...
- **Confidence**: ...
- **Watch**: ...

---

## Cross-Framework Contradictions

| Contradiction | Framework A says | Framework B says | Resolution / Which to weight |
|---|---|---|---|
| [e.g., "NSM growth vs. cohort degradation"] | [NSM trending up] | [Retention cohorts degrading] | [Which matters more and why — e.g., "Cohort signal is more structural; NSM growth is masking mix shift"] |

---

## Instrumentation Feasibility

| Metric | Data exists? | Clean & reliable? | Timely (within cadence)? | ≥30 days history? | Status |
|---|:---:|:---:|:---:|:---:|---|
| [metric] | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | Ready / Needs work / Blocked |

[Flag any metric that fails ≥1 check. Propose measurable proxy or instrumentation investment.]

---

## Review Cadence & Ownership

| Metric Level | Review Cadence | Owner | Escalation Trigger |
|---|---|---|---|
| NSM | Monthly | [exec] | [e.g., ">10% decline sustained 2 months"] |
| L1 | Weekly | [PM] | [e.g., "Misses target 3 consecutive weeks"] |
| L2 | Daily | [feature team] | [e.g., "Alert threshold crossed"] |
| Experiments | Per-experiment | [PM + data] | [e.g., "Guardrail violated"] |
| Quarterly health review | Every 13 weeks | [owner] | [e.g., "Any proxy-outcome r < 0.5"] |

---

## Assumption Registry

| # | Assumption | Framework it underpins | Confidence | Evidence | What would invalidate this |
|---|---|---|---|---|---|
| 1 | | | H/M/L | (TX) | |
| 2 | | | H/M/L | (TX) | |
| 3 | | | H/M/L | (TX) | |

---

## Adversarial Self-Critique

**Weakness 1: [Title]**
[What assumption is being made? What evidence would disprove it? Is there a scenario where this metric system is catastrophically wrong? Link to a Watch Indicator.]

**Weakness 2: [Title]**
[Same depth]

**Weakness 3: [Title]**
[Same depth]

---

## Revision Triggers

| Trigger | What to re-assess | Timeline |
|---|---|---|
| NSM-outcome correlation drops below r = 0.5 | NSM selection | Next quarterly review |
| Leading indicator no longer predicts lagging | Leading/lagging pairs | Immediate investigation |
| Assumption Registry item invalidated | Dependent framework sections | Within 1 week |
| Retention curve shape changes | PMF assessment + cohort design | Next quarterly review |
| Counter-metric threshold crossed 2+ consecutive periods | Primary metric + gaming detection | Immediate investigation |

---

## Sources

[All sources cited in the framework, with evidence tier and date.]

Rules for using this template:

1. Do not skip sections. If a section isn't applicable (e.g., HEART skipped in Step 0), write "Skipped — [reason]" and move on.
2. Every table cell with a target, benchmark, or claim must have an evidence tier tag — (T1) through (T6).
3. Section headers are conclusions, not labels. Replace generic headers (e.g., "Retention Analysis") with insight headers (e.g., "Day-30 Retention Is Masking a Cohort Degradation Problem") after completing the section.
4. The Executive Summary is written last but appears first. Do not write it until all sections are complete.
5. Instrumentation Feasibility is not optional. A beautiful metric that can't be measured is a planning artifact, not a measurement framework.

Domain Frameworks

This section IS the knowledge weapon. Each framework is encoded with its scoring rubrics, decision tables, and output format templates — not merely referenced. A PM using this skill designs metric systems that resist gaming, detect problems early, and validate causally; without these frameworks, the output degrades to a KPI list.

Framework 1: North Star Framework + Metric Decomposition Tree

The foundational architecture for metric systems. A North Star Metric (NSM) captures the core value your product delivers. The decomposition tree breaks it into actionable layers.

NSM Selection Rubric:

Score every candidate NSM against these 5 criteria. A viable NSM scores ≥4/5:

Decision point: If your best candidate scores 3/5, strengthen it before proceeding. If multiple candidates score 5/5, pick the one closest to the value moment — the instant the user gets what they came for.

Metric Decomposition Tree — 4 Levels:

North Star Metric (NSM)
├── L1 Metrics (3-5) — Exec/PM-owned, reviewed weekly
│   ├── L2 Metrics (2-4 per L1) — Feature-team-owned, reviewed daily
│   │   └── Input Metrics — Directly manipulable by shipping code
│   ├── L2 Metrics
│   │   └── Input Metrics
│   └── L2 Metrics
│       └── Input Metrics
├── L1 Metrics
│   └── ...
└── L1 Metrics
    └── ...

Decomposition Method:

1. Start with the NSM. Ask: "What 3-5 things must ALL be true for this metric to grow?"
2. For each L1, ask: "What 2-4 sub-components drive this L1?" These are L2 metrics.
3. For each L2, ask: "What can an engineer ship this sprint to move this?" Those are input metrics.
4. Validation check: Can you trace from every input metric back up to the NSM with a clear causal story? If not, the tree has a broken branch.

Metric-to-Team Ownership Mapping:

GSM (Goals-Signals-Metrics) Integration:

Before decomposing, validate the NSM using Google's GSM framework:

Key insight: The signal is the hardest part. Most PMs jump from Goal to Metric and skip Signal entirely — which is how you end up measuring pageviews when you meant to measure collaboration.

Output format — Metric Hierarchy Table:

| Level | Metric | Owner | Cadence | Target | Counter-Metric |
|-------|--------|-------|---------|--------|----------------|
| NSM   | Weekly active teams completing a project | CPO | Monthly | 50K | Quality score ≥ 4.0 |
| L1    | Activation rate (day-7) | PM, Growth | Weekly | 40% | Time-to-activate ≤ 3 days |
| L1    | Week-4 retention | PM, Core | Weekly | 65% | Session depth ≥ 3 actions |
| L2    | Onboarding completion | PM, Onboarding | Daily | 70% | Support tickets from new users |
| L2    | Invite-sent rate | PM, Virality | Daily | 25% | Invite-spam complaints |
| Input | Onboarding step-3 conversion | Eng | Per-deploy | 85% | Error rate < 1% |

Framework 2: Leading vs. Lagging Indicator Design

Every metric has a temporal character. Lagging metrics tell you what already happened. Leading metrics tell you what's about to happen. A measurement framework without leading indicators is a rearview mirror — you'll see the cliff after you've driven off it.

Temporal Lag Classification:

The Lag-to-Cadence Rule: Your monitoring cadence must match the metric's lag category. Checking a structural metric daily creates noise anxiety. Checking an immediate metric monthly means you miss fires.

Activation Metric Design (the "Aha Moment" Protocol):

The activation metric is the single most valuable leading indicator — it predicts long-term retention from early behavior.

Critical warning — Step 6 is where most teams fail. Correlation (people who invite retain better) ≠ causation (making people invite causes retention). The user who invites may simply be more engaged. You MUST run the causal experiment before declaring this your activation metric.

Leading Indicator Discovery Protocol:

For every lagging metric in your hierarchy, identify 2-3 leading indicators using this template:

| Lagging Metric | Leading Indicator | Lag Time | Correlation | Causal? | Threshold |
|---------------|-------------------|----------|-------------|---------|-----------|
| Monthly churn  | Support ticket vol | 2-3 weeks | r=0.72 | Likely (test) | >2x baseline = alert |
| Monthly churn  | Login frequency drop | 1-2 weeks | r=0.68 | Yes (validated) | <50% of avg = alert |
| Revenue growth | Pipeline velocity | 4-6 weeks | r=0.81 | Yes | <80% of target = alert |

Framework 3: Counter-Metric Design & Goodhart's Law

"When a measure becomes a target, it ceases to be a good measure." — Charles Goodhart

Every PM knows the quote. Almost none can name the four mechanisms by which it happens, or design metric systems that resist them.

Goodhart's Law — The 4 Variants (Manheim & Garrabrant):

Counter-Metric Pairing Template:

For every primary metric, complete this row:

| Primary Metric | What Could Go Wrong | Counter-Metric | Threshold | Review |
|---------------|--------------------|--------------------|-----------|--------|
| Signup rate | Fake accounts, low-quality signups | Day-7 activation rate | Must stay ≥ 35% | Weekly |
| Feature adoption | Forced adoption, dark patterns | User satisfaction (CSAT) | Must stay ≥ 4.0/5 | Bi-weekly |
| Resolution time | Premature ticket closure | Reopen rate | Must stay ≤ 8% | Weekly |
| Revenue per user | Price increases erode retention | 90-day retention | Must stay ≥ 70% | Monthly |
| DAU | Notification spam drives opens | Session duration, uninstall rate | Duration ≥ 3min, uninstall ≤ 2% | Weekly |

Guardrail Metric Protocol:

Guardrails are the non-negotiables — metrics that must NOT degrade when you optimize a primary metric. Every experiment must declare guardrails before launch.

Gaming Detection Patterns:

Quarterly Metric Health Review:

Every quarter, review each metric in the hierarchy:

Framework 4: Experiment Design

Not every question needs an experiment. Not every experiment should be an A/B test. This framework encodes when to experiment, what type to use, and how to design it properly.

Experiment Type Decision Table:

A/B Test Design Protocol:

Complete every field before launching. Incomplete fields = incomplete experiment.

| Field | Value | Notes |
|-------|-------|-------|
| Hypothesis | [If we change X, metric Y will improve by Z%] | Must be falsifiable |
| Primary metric | [ONE metric] | Pre-declared. This is what decides ship/no-ship |
| Secondary metrics | [2-3 metrics] | Exploratory. Bonferroni-corrected if used for decisions |
| Guardrail metrics | [1-3 metrics] | Must NOT degrade. Pre-set thresholds |
| MDE | [X%] | Smallest effect worth detecting (business decision, not stats) |
| Significance level (α) | [0.05 typical] | False positive tolerance |
| Power (1-β) | [0.80 minimum] | False negative tolerance |
| Sample size (N) | [computed] | Use sample_size_calculator.py |
| Duration | [X days/weeks] | Must cover full weekly cycle (min 7 days) |
| Randomization unit | [user / session / org / region] | Must match analysis unit |
| Exclusions | [who is excluded and why] | New users only? Existing only? |
| Decision rule | [Ship if primary ≥ MDE AND guardrails hold] | Written before launch |

When NOT to Experiment — Checklist:

Experiment Quality Rubric:

Score each experiment 0-6. Below 4 = do not trust results.

Framework 5: Statistical Validity for PMs

This section explains statistical concepts in plain English. No formulas. No textbook definitions. The computation scripts handle the math — this section builds the intuition to use them correctly.

Significance (α) — "How sure are you this isn't a fluke?"

Imagine flipping a coin 10 times and getting 7 heads. Suspicious? Maybe. Now imagine 100 flips and 70 heads. Very suspicious. Statistical significance measures whether your experiment result is suspicious enough to believe.

• α = 0.05 means: "I'm accepting a 5% chance this result is a coincidence." One in twenty experiments will show a "win" that's actually noise.
• Lower α (0.01) = more cautious. Fewer false alarms, but you'll miss some real effects.
• Higher α (0.10) = more aggressive. Catch more real effects, but more false alarms too.
• PM rule of thumb: Use 0.05 for most tests. Use 0.01 for high-stakes, irreversible decisions (pricing, major redesigns). Use 0.10 for quick directional learnings you'll validate later.

Power (1-β) — "Would you even notice if something real happened?"

Power is like a metal detector's sensitivity. A cheap detector (low power) misses gold coins buried 6 inches deep. A good detector (high power) finds them.

• Power = 0.80 means: "If there IS a real effect, I have an 80% chance of detecting it." Twenty percent of the time, a real improvement goes undetected.
• Low power is the silent killer. You run a test, see "no significant result," and conclude "the feature doesn't work." But maybe it DOES work — your test just wasn't powerful enough to detect it.
• What kills power: Small sample size, small effect, high variance in the metric.
• PM rule of thumb: Never run a test with power below 0.80. If your calculator says you need 100K users and you have 10K, don't run the test — you'll learn nothing.

MDE (Minimum Detectable Effect) — "What's the smallest win worth knowing about?"

Before every test, decide: "What's the smallest improvement that would change my decision?"

• If a 0.5% conversion lift isn't worth the eng effort to maintain, don't design a test to detect 0.5%. Set MDE at 2% or 5%.
• Smaller MDE = bigger sample size = longer test. This is a business tradeoff, not a statistics one.
• The conversation to have: "Are we willing to run this test for 6 weeks to detect a 1% lift? Or would a 3% lift be the threshold for shipping, in which case we need only 2 weeks?"

Confidence Intervals — "Where does the truth probably live?"

A p-value tells you "significant or not" — binary. A confidence interval tells you "the real effect is probably between X% and Y%."

• Result: "+3.2% conversion, 95% CI [0.8%, 5.6%]" means the true effect is probably between 0.8% and 5.6%. The point estimate is 3.2%, but it could be as low as 0.8%.
• PM decision rule: Look at the bottom of the confidence interval. If the worst case (0.8%) is still worth shipping, ship confidently. If the worst case crosses zero or goes negative, the result is fragile.
• Always report CIs, not just p-values. A "significant" result with CI [0.01%, 6.0%] is very different from [2.5%, 3.5%] — same p-value, wildly different confidence.

Sample Size — "How long do we need to run this?"

Sample size is determined by three inputs: your baseline rate, your MDE, and your α/power choices. It is not a guess.

• Use sample_size_calculator.py with your inputs. It outputs the required N per variant and estimated duration

…(truncated)