Product Analytics

Product analytics

A metric is a claim about the world. Most disputed numbers are not wrong arithmetic — they are two people using the same word for two different denominators, or a dashboard built on an event that was never emitted on the path that matters.

When this fires

Someone is about to define, report or argue about a product number: activation, conversion, retention, funnel drop-off, adoption of a feature. It also fires before instrumentation is added, which is the cheapest moment to get it right. It does not fire when the metric definition is already settled and the task is only to compute or chart it.

Four states, not one

Keep these apart in every sentence you write, because conflating them is how a metric that does not exist ends up on a slide:

• Defined — written down with numerator, denominator, population, window.
• Instrumented — the emitting code exists on every path that should count.
• Collecting — events are arriving in the destination, from real traffic, since a known date.
• Reconciled — the number has been checked against an independent source and agrees.

Only the fourth is a number you may quote without a caveat.

Procedure

1. Name the decision the metric serves. Who changes what behaviour at which threshold. If no answer, say so — a metric nobody will act on is not worth the instrumentation cost, and saying that is a legitimate result of this procedure.
2. Write it as a rate with both halves visible. Numerator, denominator, the population each is drawn from, and the time window. "Activation rate" is not a definition. "Accounts created in week W that reached first successful import within 7 days / accounts created in week W" is.
3. Fix the unit of aggregation — user, account, session, device, request — and use the same unit on both halves. A per-session numerator over a per-user denominator is the most common silent error in this whole area, and it never announces itself.
4. Specify each event precisely: the name, the exact moment it fires, the properties carried, and the near-miss cases it deliberately excludes (retries, server-side replays, internal or test accounts, bot traffic). Write the exclusions down; they are the part that gets lost.
5. Find the emitting code before believing the event exists. Search the repository for the event name and read every call site. Check the paths nobody remembers: the mobile client, the API-only path, the error branch, the redirect that returns early. Absent call site means the metric is defined, not instrumented — report that, do not estimate around it.
6. For a funnel, pin the entry cohort once and carry it through. Decide and record: must the steps occur in order; what attribution window each step has; are users who enter twice counted twice. A funnel whose steps each have their own population is four unrelated numbers in a row.
7. Add a guardrail or counter-metric. Any metric that can be moved by degrading something else — speed at the cost of errors, signups at the cost of retention — gets the opposing metric defined alongside it, or it will be gamed by accident.
8. State the trustworthy-from date. The first date the current definition and the current instrumentation both held. Before that date the series is a different metric wearing the same name; do not chart across the boundary without marking it.
9. Reconcile against something independent before publishing: a database count, billing records, a manual count for one day, the previous system. Report the gap as a number and its likely cause. Agreement within a stated tolerance is the evidence; "looks about right" is not.
10. Stop and ask before anything outward-facing. Adding tracking to production, changing the semantics of an event other dashboards already use, publishing a dashboard others will act on, or collecting a new property about people. Especially: never add personal data — email, name, free-text input, precise location, anything identifying — to event properties on your own judgement. Propose it, name the field, and wait.

Checklist

• The decision this metric informs is written down, or its absence is flagged
• Numerator, denominator, population and window are all stated
• Both halves use the same unit of aggregation
• Every event's firing moment, properties and exclusions are specified
• Each event was located in the emitting code, on every path that should count
• Funnel entry cohort, step ordering and attribution windows are fixed
• A guardrail metric exists for anything that can be gamed
• The trustworthy-from date is stated
• The number was reconciled against an independent source, with the gap quoted
• Any new tracking or new property was proposed, not added

Failure handling

• Two sources disagree — do not average them and do not pick the friendlier one. Take one narrow slice (one day, one account) and trace it through both until the divergence is a named cause: a missing call site, a timezone, a deduplication rule, a filter on one dashboard.
• The event is not instrumented — the finding is "not measurable today", with the list of paths that would need to emit. An estimate derived from a proxy is a proxy: label it as one, every time it is quoted.
• The event fires more than once per real occurrence — say whether the metric counts events or distinct units, and fix the definition rather than the data.
• Historical data does not go back far enough — report the window you actually have. Never backfill by assumption and present the result as history.
• The definition already exists somewhere — use it, or change it deliberately and say so. A second definition of an existing name is worse than a bad definition of a new one.
• Personal or sensitive data appears in event properties — stop, name the field and where it is emitted, and raise it. Do not quietly keep querying it.

Evidence to report

The metric written out in full — numerator, denominator, population, window, unit. The file paths and call sites where each event is emitted, and the paths found to be missing it. The exclusion list. The trustworthy-from date. The reconciliation: both numbers, the gap, the explanation. And the parts still unmeasurable, named as such rather than approximated into the table.