Data Analytics Metric Definition

Metric definition contract

Purpose

Two teams report "active users" and the numbers differ by 18%. Nobody is wrong; they are answering different questions. The argument that follows is never about the data — it is about a definition that was never written down, so each analyst made reasonable choices about timezone, deduplication, and what counts as activity. This skill produces the contract that ends that argument, and the restatement rules that stop the definition drifting again silently.

Prerequisites

• Inputs: the metric name as currently used; the decision it is supposed to support; every existing implementation of it (queries, dashboards, reports) with the numbers each currently produces; the source tables and their event semantics.
• Access: the systems that produce the underlying events, and the person with authority to accept a definition on behalf of the business.

If the two conflicting numbers cannot both be reproduced, stop. A definition written without seeing where existing implementations diverge will just become a third number.

Procedure

1. Write the decision the metric serves, in one sentence, before defining anything. "We use this to decide whether to keep investing in the onboarding flow." Definitions have no correct answer in the abstract; they have a correct answer relative to a decision. A metric with no decision attached is a number the organisation maintains for free and should probably retire.

2. Fill every row of the contract. A blank row is where the next disagreement will come from.

   Element	The question it settles	Failure if left blank
   Population	Which entities are eligible at all? Internal, test, bot, deleted, and trial entities in or out?	Test traffic inflates every figure
   Qualifying event	What exactly must happen to count? Which event names, which statuses?	"Activity" means login to one team and any API call to another
   Grain	Per what — user, account, session, transaction?	Multi-user accounts double or halve the number
   Time window	Rolling N days, or a calendar period?	Rolling-28 and calendar-month never agree
   Timezone and boundary	Which timezone defines the day boundary? Event time or ingestion time?	Reports diverge by roughly one day's volume
   Deduplication	One entity acting five times counts as how many?	Counts vs distinct counts differ by an order of magnitude
   Exclusions	Refunds, reversals, cancellations, internal transfers, fraud	Revenue metrics overstate systematically
   Late/backdated data	How long does the window stay open, and does the past change?	Yesterday's number changes and nobody knows why
   Null and unknown handling	Excluded, bucketed, or imputed?	Segment sums stop matching the total
   Source of record	The one table or model that is authoritative	Two pipelines drift apart quietly
   Owner	A named person who accepts changes	Nobody can resolve the next dispute

3. Resolve the divergences you found in step 0, explicitly, and record the ones you rejected. For each way the existing implementations differed, state which choice the contract makes and why. The rejected alternatives matter as much as the chosen one — they are what stops the argument reopening.

4. Decide restatement policy up front.

   Policy	Behaviour	Use when
   Frozen	Once a period closes, the number never changes	External or board reporting, anything with a compliance trail
   Open window	Restates for N days, then freezes	Operational metrics with known late-arriving data
   Always current	Recomputed from source every run	Internal exploration only

   A metric that is Frozen for the board and Always current on the dashboard is two metrics and must be named as two. Naming them the same thing guarantees a meeting.

5. Define the segment rule. State the dimensions the metric may be split by and confirm segments sum to the total. If they do not — because entities can be in two segments, or a segment is unknown — say so in the contract. Silent non-additivity is one of the most common causes of "the numbers don't add up".

6. Set a tolerance and a reconciliation partner. Name the other figure this metric must agree with (the finance ledger, the billing export, the source system's own report) and the acceptable difference. A metric with no external check drifts undetected. If the tolerance is exceeded, the metric is treated as broken and its consumers are told — not quietly patched.

7. Version the contract, and treat a change as a breaking change. Any change to population, qualifying event, or exclusions changes the number's history. Publish: what changed, the effective date, whether history was restated, and the size of the step. A metric that changes definition without a published step is how a trend chart lies for a year.

8. Publish the plain-language sentence alongside the specification. "A user who performed at least one qualifying action in the trailing 28 days, counted once, in UTC, excluding internal and test accounts." That sentence goes in the dashboard tooltip. Contracts nobody reads are only slightly better than no contract; the one-line version is what actually propagates.

Worked example of the divergence this prevents

Two teams report monthly active users:

Neither number is wrong; there was no contract. The reconciliation is done by walking these rows, not by re-running the queries.

Data handling

Classification: Internal; the contract itself is usually shareable, but worked examples must not carry real customer identifiers or account numbers — use synthetic values. If the metric covers personal, financial, or regulated data, state that in the contract so downstream users know the outputs inherit the classification. If someone pastes real customer records or positions into the conversation while defining the metric, flag it and do not proceed until removed.

Boundaries

• A specific report or dashboard needs verification before it is circulated — data-analytics-report-qa.
• The query implementing the definition needs correctness or cost review — data-analytics-sql-review.
• The metric is an experiment's success measure and the question is whether the result is real — that is experiment readout, not definition; define the metric here first, then hand off.
• Choosing where the metric is computed and stored (warehouse model, streaming pipeline) is an architecture decision — engineering-decision-record.

Hand-offs

• Receives from: data-analytics-report-qa (reconciliation failures that turn out to be definitional, not computational).
• Routes to: data-analytics-sql-review (implementation of the contract); engineering-decision-record (when the contract forces a pipeline or storage choice).