Product Analytics
Define metrics that matter. Design experiments that produce valid insights. Measure feature adoption and business impact. Make data-informed product decisions.
Announce at start: "I'm using the product-analytics skill to [purpose]."
Checklist
You MUST create a task for each of these items and complete them in order:
- Define the North Star metric — What single metric captures your product's core value?
- Map the user lifecycle (AARRR) — Where are the biggest opportunities?
- Define feature adoption framework — How will you measure if features succeed?
- Design experiments (A/B tests) — If testing, follow the full process
- Set up measurement before building — Instrument before development
- Define kill criteria — When to deprecate features that don't work
Step 1: Define the North Star Metric
The single metric that captures the core value your product delivers — a leading indicator of long-term success that expresses value to users, not just revenue.
| Company |
North Star Metric |
Core Value |
| Spotify |
Time spent listening |
Music discovery and enjoyment |
| Airbnb |
Nights booked |
Travel accommodations |
| Slack |
Messages sent per team |
Team communication |
| Figma |
Weekly active editors |
Collaborative design |
| Stripe |
Payment volume processed |
Payment infrastructure |
How to define yours: Identify the core action that delivers value. Determine how frequently users must take it for ongoing value. Ensure it's measurable and hard to game.
Input metrics (what teams influence to move the North Star): e.g., activation rate → onboarding team, retention rate → core experience team, collaboration invites → growth team, time-to-value → performance team.
Step 2: Map the User Lifecycle (AARRR)
| Stage |
Question |
Key Metrics |
| Acquisition |
How do users find you? |
Traffic sources, conversion by channel, CAC |
| Activation |
Do they have a great first experience? |
Sign-up rate, onboarding completion, time to "aha moment" |
| Retention |
Do they come back? |
DAU/MAU, churn rate, D1/D7/D30 retention |
| Referral |
Do they tell others? |
NPS, viral coefficient, referral sign-ups |
| Revenue |
How do you make money? |
ARPU, LTV, MRR/ARR, conversion to paid |
Diagnosis: Find the bottleneck stage → form a hypothesis → design an intervention → measure impact.
| Pattern |
Likely Issue |
Focus |
| High acquisition, low activation |
Onboarding broken or expectations mismatch |
Activation |
| High activation, low retention |
Core value not sticky enough |
Retention |
| High retention, low referral |
Product good but not remarkable |
Delight |
| High everything, low revenue |
Monetization strategy needs work |
Pricing |
Step 3: Feature Adoption Measurement
Adoption funnel: Exposed → Engaged → Activated → Retained → Power User
| Stage |
Definition |
Metric |
| Exposed |
Saw the feature exists |
% of target users who viewed it |
| Engaged |
Interacted once |
% who clicked/opened/tried |
| Activated |
Got value from first use |
% who completed the key action |
| Retained |
Came back |
% who used it again within N days |
| Power User |
Core workflow habit |
% using at least X times/week |
| Metric |
Definition |
Target |
| Adoption Rate |
% of target users who use the feature |
Depends on feature type |
| Time to Adopt |
How long after exposure until first use |
< 7 days for promoted features |
| Feature Retention |
% of first-time users who return |
> 40% at D7 is healthy |
| Feature Stickiness |
DAU/MAU for the feature |
> 20% is sticky |
| Cannibalization |
Does new feature reduce usage of existing ones? |
Monitor for displacement |
Always segment adopters by persona, account age, plan/tier, region, and device/platform.
Step 4: Design A/B Tests
Process:
- Form a hypothesis: "If we change [X] to [Y], we will see [Z impact] because [rationale]"
- Define primary metric: ONE metric that determines winner/loser
- Define guardrail metrics: Metrics that must NOT degrade (e.g., revenue per user ≤ 2% drop)
- Calculate sample size: Based on baseline rate, minimum detectable effect, 80% power, 95% significance
- Randomize and run: Random split, pre-calculated duration (1–4 weeks). Do NOT peek early.
- Analyze: Check statistical significance (p < 0.05), practical significance, guardrails, segment results
- Decide: Ship winner, iterate if inconclusive, discard if negative
Pitfalls:
| Pitfall |
Prevention |
| Peeking (stopping early) |
Pre-commit to duration, don't look until over |
| Too many variants |
Test 2–3 max |
| Too many metrics (p-hacking) |
Pre-register one primary metric |
| Novelty effect |
Run at least 2–4 weeks |
| Small sample |
Calculate required size upfront |
| Ignoring segments |
Always segment results |
| No guardrail metrics |
Always define what must not degrade |
When NOT to A/B test: Sample too small for significance, bug fix/infra change (just ship), compliance requirement, very small user segment, experiment cost exceeds learning value, can't measure outcome reliably.
Step 5: Instrument Before Building
Before launch, define for every feature:
| Artifact |
Contents |
| Success metrics |
Metric, definition, events, target (e.g., adoption rate = feature_engaged / total users > 30%) |
| Events to track |
Event name, when fired, properties, example values |
| User properties |
Property name, type, example (e.g., plan_tier: String, team_size: Integer) |
Before launch, verify: all events fire correctly in staging, event properties populated, data flows to your analytics tool, dashboard is built with test data, key metrics are queryable.
Step 6: Kill Criteria
| Timeline |
Metric |
Threshold |
Action |
| 30 days |
Adoption rate |
< 20% of target users |
Investigate discoverability or value |
| 60 days |
Feature retention |
< 25% return |
Consider redesign or deprecation |
| 90 days |
Adoption rate |
< 10% of target users |
Deprecate |
| Ongoing |
Support tickets |
> 20% of total |
Evaluate cost vs. benefit |
Key Principles
- Metrics should drive decisions — If a metric doesn't change what you do, stop tracking it.
- Measure outcomes, not outputs — "30% adopted" not "we shipped 3 features."
- Instrument before building — You can't retroactively add tracking.
- Segment everything — Averages lie. Always look at segments.
- One primary metric per experiment — Multiple metrics = multiple comparison problem.
- Define kill criteria before launch — Don't let zombie features accumulate.
- Qualitative + quantitative — Data tells you WHAT. User research tells you WHY.
- North Star is a compass — Guide decisions, not just measure past performance.
Common Mistakes
- Tracking vanity metrics (page views, downloads) instead of actionable metrics (activation, retention)
- No defined success metrics before shipping; instrumentation as an afterthought
- A/B testing everything (some things should just ship)
- Ignoring practical significance (statistically significant but too small to matter)
- Over-optimizing for one metric at expense of others; not segmenting results
- Keeping features alive because removing them is awkward (use kill criteria)
Key References
- "Lean Analytics" by Alistair Croll and Benjamin Yoskovitz
- "Hacking Growth" by Sean Ellis and Morgan Brown
- Amplitude's "The North Star Playbook"
- Dave McClure's AARRR framework (500 Startups)
- Reforge's experimentation and growth programs
1---2name: product-analytics3description: Use when defining product metrics, designing experiments, analyzing feature adoption, or setting up measurement frameworks.4---56# Product Analytics78Define metrics that matter. Design experiments that produce valid insights. Measure feature adoption and business impact. Make data-informed product decisions.910**Announce at start:** "I'm using the product-analytics skill to [purpose]."1112## Checklist1314You MUST create a task for each of these items and complete them in order:15161. **Define the North Star metric** — What single metric captures your product's core value?172. **Map the user lifecycle (AARRR)** — Where are the biggest opportunities?183. **Define feature adoption framework** — How will you measure if features succeed?194. **Design experiments (A/B tests)** — If testing, follow the full process205. **Set up measurement before building** — Instrument before development216. **Define kill criteria** — When to deprecate features that don't work2223## Step 1: Define the North Star Metric2425The single metric that captures the core value your product delivers — a leading indicator of long-term success that expresses value to users, not just revenue.2627| Company | North Star Metric | Core Value |28|---------|-------------------|------------|29| Spotify | Time spent listening | Music discovery and enjoyment |30| Airbnb | Nights booked | Travel accommodations |31| Slack | Messages sent per team | Team communication |32| Figma | Weekly active editors | Collaborative design |33| Stripe | Payment volume processed | Payment infrastructure |3435**How to define yours:** Identify the core action that delivers value. Determine how frequently users must take it for ongoing value. Ensure it's measurable and hard to game.3637**Input metrics** (what teams influence to move the North Star): e.g., activation rate → onboarding team, retention rate → core experience team, collaboration invites → growth team, time-to-value → performance team.3839## Step 2: Map the User Lifecycle (AARRR)4041| Stage | Question | Key Metrics |42|-------|----------|-------------|43| **Acquisition** | How do users find you? | Traffic sources, conversion by channel, CAC |44| **Activation** | Do they have a great first experience? | Sign-up rate, onboarding completion, time to "aha moment" |45| **Retention** | Do they come back? | DAU/MAU, churn rate, D1/D7/D30 retention |46| **Referral** | Do they tell others? | NPS, viral coefficient, referral sign-ups |47| **Revenue** | How do you make money? | ARPU, LTV, MRR/ARR, conversion to paid |4849**Diagnosis:** Find the bottleneck stage → form a hypothesis → design an intervention → measure impact.5051| Pattern | Likely Issue | Focus |52|---------|-------------|-------|53| High acquisition, low activation | Onboarding broken or expectations mismatch | Activation |54| High activation, low retention | Core value not sticky enough | Retention |55| High retention, low referral | Product good but not remarkable | Delight |56| High everything, low revenue | Monetization strategy needs work | Pricing |5758## Step 3: Feature Adoption Measurement5960**Adoption funnel:** Exposed → Engaged → Activated → Retained → Power User6162| Stage | Definition | Metric |63|-------|-----------|--------|64| **Exposed** | Saw the feature exists | % of target users who viewed it |65| **Engaged** | Interacted once | % who clicked/opened/tried |66| **Activated** | Got value from first use | % who completed the key action |67| **Retained** | Came back | % who used it again within N days |68| **Power User** | Core workflow habit | % using at least X times/week |6970| Metric | Definition | Target |71|--------|-----------|--------|72| **Adoption Rate** | % of target users who use the feature | Depends on feature type |73| **Time to Adopt** | How long after exposure until first use | < 7 days for promoted features |74| **Feature Retention** | % of first-time users who return | > 40% at D7 is healthy |75| **Feature Stickiness** | DAU/MAU for the feature | > 20% is sticky |76| **Cannibalization** | Does new feature reduce usage of existing ones? | Monitor for displacement |7778Always segment adopters by persona, account age, plan/tier, region, and device/platform.7980## Step 4: Design A/B Tests8182**Process:**831. **Form a hypothesis:** "If we change [X] to [Y], we will see [Z impact] because [rationale]"842. **Define primary metric:** ONE metric that determines winner/loser853. **Define guardrail metrics:** Metrics that must NOT degrade (e.g., revenue per user ≤ 2% drop)864. **Calculate sample size:** Based on baseline rate, minimum detectable effect, 80% power, 95% significance875. **Randomize and run:** Random split, pre-calculated duration (1–4 weeks). Do NOT peek early.886. **Analyze:** Check statistical significance (p < 0.05), practical significance, guardrails, segment results897. **Decide:** Ship winner, iterate if inconclusive, discard if negative9091**Pitfalls:**9293| Pitfall | Prevention |94|---------|-----------|95| **Peeking** (stopping early) | Pre-commit to duration, don't look until over |96| **Too many variants** | Test 2–3 max |97| **Too many metrics** (p-hacking) | Pre-register one primary metric |98| **Novelty effect** | Run at least 2–4 weeks |99| **Small sample** | Calculate required size upfront |100| **Ignoring segments** | Always segment results |101| **No guardrail metrics** | Always define what must not degrade |102103**When NOT to A/B test:** Sample too small for significance, bug fix/infra change (just ship), compliance requirement, very small user segment, experiment cost exceeds learning value, can't measure outcome reliably.104105## Step 5: Instrument Before Building106107**Before launch, define for every feature:**108109| Artifact | Contents |110|----------|----------|111| **Success metrics** | Metric, definition, events, target (e.g., adoption rate = `feature_engaged` / total users > 30%) |112| **Events to track** | Event name, when fired, properties, example values |113| **User properties** | Property name, type, example (e.g., `plan_tier`: String, `team_size`: Integer) |114115**Before launch, verify:** all events fire correctly in staging, event properties populated, data flows to your analytics tool, dashboard is built with test data, key metrics are queryable.116117## Step 6: Kill Criteria118119| Timeline | Metric | Threshold | Action |120|----------|--------|-----------|--------|121| 30 days | Adoption rate | < 20% of target users | Investigate discoverability or value |122| 60 days | Feature retention | < 25% return | Consider redesign or deprecation |123| 90 days | Adoption rate | < 10% of target users | Deprecate |124| Ongoing | Support tickets | > 20% of total | Evaluate cost vs. benefit |125126## Key Principles127128- **Metrics should drive decisions** — If a metric doesn't change what you do, stop tracking it.129- **Measure outcomes, not outputs** — "30% adopted" not "we shipped 3 features."130- **Instrument before building** — You can't retroactively add tracking.131- **Segment everything** — Averages lie. Always look at segments.132- **One primary metric per experiment** — Multiple metrics = multiple comparison problem.133- **Define kill criteria before launch** — Don't let zombie features accumulate.134- **Qualitative + quantitative** — Data tells you WHAT. User research tells you WHY.135- **North Star is a compass** — Guide decisions, not just measure past performance.136137## Common Mistakes138139- Tracking vanity metrics (page views, downloads) instead of actionable metrics (activation, retention)140- No defined success metrics before shipping; instrumentation as an afterthought141- A/B testing everything (some things should just ship)142- Ignoring practical significance (statistically significant but too small to matter)143- Over-optimizing for one metric at expense of others; not segmenting results144- Keeping features alive because removing them is awkward (use kill criteria)145146## Key References147148- "Lean Analytics" by Alistair Croll and Benjamin Yoskovitz149- "Hacking Growth" by Sean Ellis and Morgan Brown150- Amplitude's "The North Star Playbook"151- Dave McClure's AARRR framework (500 Startups)152- Reforge's experimentation and growth programs
guhcostan