Data Analysis

Data Analysis

The number is the smallest part of the answer. The definition, the denominator, the sanity check, and the caveat are what make it safe for somebody to act on.

Standing reference: references/data-quality-checklist.md.

Process

1. Restate the question and the decision behind it. "Did signups drop" and "should we roll back last night's release" need different analyses and different precision. Without the decision you cannot tell when the analysis is finished.

2. Define the metric in writing before you query. Numerator, denominator, time grain, time zone, exclusions: internal accounts, bots, refunds, test records. Most disputed numbers are two people using one word for two definitions, discovered an hour into an argument.

3. Verify the data before trusting it. Freshness for the period, row volume against the prior period, null rates on the columns you filter on. A pipeline gap is indistinguishable from a real drop until you check, and it is the more common cause.

4. Reconcile against something already believed. Billing, the previous dashboard, a hand count for one day. Quote both numbers. An unreconciled number is a draft.

5. Segment before concluding. A flat aggregate hides two populations moving in opposite directions, and the interesting finding is almost always inside a segment. Report sample size with every segment figure.

6. Rule out the boring explanations, in this order: a data problem, seasonality or day of week, a release, a tracking or instrumentation change, one large account, reporting lag. Name which you tested and how, including the ones that survived.

7. Separate correlation from cause, in the sentence itself. Observational comparisons between people who did and did not do something are selection effects until an experiment or a natural one says otherwise. Write "associated with", and say what would be needed to claim more.

8. State the caveat with the number, not underneath it. Sample size, window sensitivity, what would change the conclusion. A finding that only holds for one window is a finding about the window.

Experiment readouts

• The metric, the exposure definition, and the analysis window are fixed before looking.
• Report the absolute effect and the interval, not only whether it cleared a threshold.
• A non-significant result is a result. Report it as "no detectable effect above ", never as "no effect".
• Check the guardrail metrics and the sample ratio before reading the headline number.

Output

## Question: <what was asked, and the decision it feeds>

- **Metric:** <numerator> / <denominator>, <grain>, timezone <tz>, excluding <list>
- **Answer:** <number, absolute and relative change, n>
- **Query:** `<path>` — source <table>, as of <date>
- **Data checks:** freshness <ok | stale>, volume vs prior <n>, nulls <n>
- **Reconciled against:** <trusted source> — <both numbers>
- **Segments:** <segment> <value> (n=<n>) ...
- **Ruled out:** <data | seasonality | release | tracking change | outlier account>
- **Caveats:** <sample size, window sensitivity, what this cannot show>
- **Confidence:** high | medium | low, and what would raise it

Verification

• Every number traces to a saved, re-runnable query.
• Every rate has its denominator stated, and every segment has its sample size.
• The result reconciles against one independent source; both numbers are quoted.
• At least one alternative explanation was tested and reported, including when it survived.
• No causal verb appears where the design does not support one.

Red flags