Data Analytics Report QA

Report QA before publication

Purpose

The expensive data failures are not wrong queries; they are silent ones. A pipeline stopped three days ago and the dashboard shows a plausible flat line. A join dropped 4% of rows and the total still looks reasonable. A segment filter excluded the largest region and nobody noticed because nobody knew what the total should be. This skill is the fixed set of checks that catch silent failure before a decision is made on the number, and the honest statement of limits that goes out with it.

Prerequisites

• Inputs: the report or figures in final form; the query, model, or extract behind them; the metric definitions in use; at least one independent source to reconcile against; the prior period's published figures.
• Runtime: ability to re-run the underlying query or inspect the pipeline's last successful run. A QA pass done by reading the output alone catches formatting errors and nothing else.

If there is no independent source to reconcile against, say so explicitly in the output rather than skipping the check silently — unreconciled is a materially different claim from reconciled.

Procedure

Run all seven. They are ordered so the cheap checks that invalidate everything come first.

1. Freshness. What is the maximum timestamp in the underlying data, and when did the pipeline last complete successfully? Compare against the expected cadence. Stale data that renders without error is the most common silent failure, and a chart of a stalled feed looks like a stable business.

2. Completeness. Row counts per period against the same periods historically. A drop or spike beyond the normal band is a defect until proven otherwise. Check the tail explicitly: the current period is usually partial, and a partial period plotted next to complete ones reads as a collapse. Either exclude it or mark it.

3. Reconciliation. Tie the headline figure to something computed by a different path — the source system's own report, the finance ledger, the billing export, a count from the operational database. Record the difference and whether it is within the agreed tolerance. Two figures from the same pipeline agreeing is not reconciliation; it is the same computation twice.

4. Additivity. Do the segments sum to the total? If not, identify why: overlapping membership, nulls in the dimension, a filter applied at one level and not another, or rounding. Every one of those is a finding; unknown residual is the worst kind, because it means the segmentation cannot be trusted at all.

5. Movement plausibility. For every figure moved more than a set threshold from the prior period, you must be able to name the cause — a real business event, a known change, or a data defect. An unexplained large move is a defect until proven otherwise. The failure this prevents is the analyst who explains a pipeline bug as a business story and is believed.

6. Definition alignment. Confirm each figure was computed with the current agreed definition, and that any definition change since the last publication is disclosed with its effective date and step size. Charts spanning a definition change without a marker are misleading even when every number is correct.

7. Filter and grain audit. Re-read the query's filters and grouping against what the report title claims. Common defects: an inner join silently acting as a filter; a date filter on the wrong column (created vs updated vs event); a deduplication that removes legitimate repeats; a leftover development limit clause.

Severity and what to do

The limits statement

Every published set of numbers carries three lines: what the data covers (period, population, source, as-of timestamp); what it does not support (the questions readers will try to answer with it and cannot); and known caveats (open reconciliation differences, partial periods, definition changes). This is the section that prevents a directional figure being used as a precise one, and it is the section most often cut for space. Do not cut it.

Failure modes this skill exists to prevent

• The plausible flat line. A stopped feed renders as stability. Caught by check 1 only; no visual inspection catches it.
• Join loss. A join drops rows and the total is still credible. Caught by checks 2 and 3.
• Filter drift. A dashboard filter left set from a previous investigation. Caught by check 7; screenshotting a filtered dashboard into a deck is how this reaches a board.
• Narrative fitting. A defect gets an interesting business explanation. Prevented by check 5's requirement to name a specific cause, verifiable independently.
• Silent republication. A corrected number is republished with no notice, so people holding the old figure never learn. Restatements are announced.

Data handling

Classification: inherits from the underlying data — treat as Confidential if the report contains personal, financial, or client-level data. Aggregate before distribution wherever the decision does not require row-level detail, and check small-cell disclosure: a segment with a handful of entities can re-identify them even in an aggregate. Never paste raw customer records, account numbers, or positions into a conversation to have them checked; work from aggregates or masked extracts. If such data is supplied, flag it and stop.

Boundaries

• Two teams disagree about what the metric means — data-analytics-metric-definition; QA cannot resolve a definitional dispute and will keep producing "reconciles to neither".
• The query itself needs correctness or cost review — data-analytics-sql-review.
• The report is an experiment readout and the question is statistical validity rather than data integrity — that needs an experiment readout procedure; these checks still apply first.
• The numbers concern an incident's customer impact for an external statement — run these checks, then hand to engineering-incident-postmortem.

Hand-offs

• Receives from: data-analytics-sql-review (a query cleared for correctness still needs these output-level checks); data-analytics-metric-definition (the contract this pass verifies compliance with).
• Routes to: data-analytics-metric-definition when reconciliation failures turn out to be definitional; data-analytics-sql-review when a check points at the query rather than the data.
• See also: it-access-review and engineering-incident-postmortem, both of which publish figures that should pass this gate before circulation.