Descriptive Evidence

Descriptive Evidence

Overview

Causal identification asks what an intervention did; structural estimation asks what a world we haven't seen would do; prediction asks what is likely true of a unit, so I can act on it. Beneath all three sits the work the discipline rushes past: describing what is actually in the data — the trend, the gap, the distribution, the summary-statistics table, the stylized facts that open almost every empirical paper. The deliverable is a faithful picture of the data — often the whole job, otherwise the thing that motivates the causal, structural, or predictive question that follows.

This is a legitimate destination, not a way-station: "just show me the trend" does not mean "skip the rigor" — it means the rigor is about the description itself.

The signature failure of this arm is the composition / aggregation artifact — a trend or gap that looks within-group but is really a shift in the mix (the aggregate wage rose because low-wage workers exited, not because anyone got a raise; a national rate can fall while it rises in every region — Simpson's paradox). It reconciles, reproduces, and plots beautifully while quietly describing the denominator or the changing sample, not the thing you named — the descriptive analog of leakage: a confident artifact.

The twin failure is the descriptive-to-causal slide: Y rises alongside X, and the write-up says "X raised Y." A described co-movement is a correlation; the moment it wears a causal verb it has left this arm for a claim it hasn't earned. A striking stylized fact's highest use is to motivate a causal question, which routes to causal-identification and earns the causal verb there — not here.

Core principle: a descriptive fact is trustworthy only when it survives the ordinary alternative explanations for the pattern — composition, deflation, scaling, selection, definition — and is stated as the correlation it is, not the cause it isn't. Everything below serves that one sentence.

What are you producing? — name the deliverable

Name your deliverable; it sets how hard each section below bites.

This is not a fork into other skills — one skill at several polish levels. The fork (causal / structural / predictive) is what you route to after the description, if a question emerges.

Where this sits

question-framing fires first (or alongside) — it pins the unit, population, decision, and joins that assemble the data. Descriptive-evidence is the execution craft once you know the output is a description: how to compute and present it so the fact is real. Joins get the same scrutiny any join does — hand them to data-contracts; a stylized fact built on a fan-out join is a fabricated fact. A fact you will publish is worth reconciling and reproducing — offer result-verification before it lands in a deck or paper; it runs on the user's ask. For a map, the split holds the same way: question-framing frames what each mark represents and which joins assemble it; descriptive-evidence makes it honest (rate-not-count, spatial unit, color breaks, the silently-dropping spatial join — see Descriptive maps). The write-up itself is econ-writing's job; this skill produces the facts, not the prose.

Fix the comparability choices — before you plot

A handful of choices silently decide what the fact says. Fix each one before you plot, and annotate it at the code site with the # why: convention (the code-level echo of a decisions log) — "indexed to 2015 dollars" six months from now is unrecoverable from a bare number:

1. Denominator. A count, a level, a rate, a share — of what? "Opioid deaths rose" (count) and "the death rate rose" (per 100k) can point opposite directions when population moves. Name the denominator and hold it fixed.
2. Real vs. nominal + base year. Any dollar figure compared across time must be deflated to a stated base year, or the "growth" is partly inflation. State the deflator (CPI, PCE, a sector index) and the base.
3. Per-capita / scaling. Compare jurisdictions, firms, or eras of different size per unit (per capita, per worker, per revenue dollar), or the biggest unit dominates every panel and the "fact" is just "California is large."
4. Weighting. An unweighted mean of a weighted sample (survey weights, value- vs. equal-weight) answers a different question. Decide which population the fact is about, weight to it, and state it.
5. Unit of observation. What is one row — a person, a person-year, a firm, a market? The same data at two grains gives two facts; a fuzzy unit is where double-counting hides.
6. Window + frequency. The sample period and frequency (monthly/quarterly/annual) change the shape; a trend that exists only because it starts in a trough or ends at a peak is an artifact of the window.
7. Aggregation level. The level at which you average is exactly where composition effects live (next section) — choosing it is choosing whether you'll see them.

These are the descriptive analog of the spec the other arms sign off, deliberately lighter. Decide and note them for a quick pass; for a fact you'll report, pin them and run the robustness check below, the way a reported effect earns its robustness battery.

The composition check — the signature discipline

This is the highest-value check in the arm — the descriptive analog of the permutation/null probe in prediction. Whenever an aggregate — a mean, a rate, a total, a gap between groups, a trend over time — moves or differs, ask one question before you believe it:

Is this a change within the groups, or a change in the mix of groups?

An aggregate can move with nothing moving inside any subgroup, purely because the weights shifted. When the number matters, decompose it rather than asserting it:

• Within-vs-between decomposition. Split the change into the within-group part (mix held fixed) and the mix part (group values held fixed). If it's mostly between, the honest sentence is about the mix, not the level.
• Shift-share for aggregates built from a changing set of components (industries, regions, products) — separate component-level change from reallocation across components.
• Oaxaca-style for decomposing a gap between two groups (a wage gap, a price gap) into a composition part and an unexplained part.
• Look inside before you average. Plot the subgroups. Simpson's paradox is invisible in the aggregate and obvious the moment you facet.

Check for selection into the sample over time, too: if who is in the data changes across the window — entry/exit, a coverage change, a shifting definition — a trend in the average can be entirely a trend in who's being averaged. A break that lines up with a coverage or definition change is a measurement artifact until proven otherwise.

If the surprise survives the decomposition, you may have a real fact; if it doesn't, you've caught the artifact before it became a headline.

A stylized fact must be robust — or it isn't stylized

The word stylized is a promise: the fact is stable, not an artifact of one arbitrary choice. A fact you'll report has to survive the obvious alternative versions of itself — a different reasonable denominator, deflator, weighting, window, or cutpoint. This is the descriptive sibling of the causal robustness battery, and it's cheap: re-cut the same fact three sensible ways and look.

If it holds across them, say so — that robustness is the credibility. If it flips with the deflator, vanishes when you drop the first year, or reverses when you weight, it is fragile, not stylized — downgrade the claim ("over 2015–2019, per-capita, …") rather than report the fragile version as general. A stylized fact that exists in only one specification is a fished fact wearing a descriptive hat.

Is the number plausible? — triangulate the level against an external anchor

The composition check and robustness re-cuts interrogate the data against itself — whether the number is computed right. They can't tell you whether it means what you named it: a count can reconcile, reproduce, and survive every alternative cut, and still be the wrong measure, because the construct it captures is only a visible slice of what you called it. That gap is reliability (computed correctly) vs. validity (measures the construct) — an arm that runs only internal checks certifies a reliable measure of the wrong quantity and never notices.

Validity is checked against something outside the dataset. When the level of a measure matters — especially when it surprises you — run at least one of these before reporting it:

• Known-shock check. Does the series move, at the right time and sign, across a launch/policy/coverage change that should move it? A measure that doesn't budge (or moves when nothing happened) is tracking something else — the descriptive cousin of a placebo.
• External benchmark. Does the level sit near an independent estimate of the same quantity? An order-of-magnitude gap is a finding to explain with a mechanism (a narrower construct, a coverage limit), not to report flat — state the benchmark and its comparability caveat.
• Alternative-construct coverage. Does the measure capture the construct, or only the slice visible in this source? Expand the definition along the dimension you suspect is missing and see how far the level moves — a level that's an artifact of where you looked is a coverage limit wearing the name of the whole.

If the level survives the anchor, say so and name it — the fact is plausible, not just robust. If it doesn't, that's a result to investigate (wrong-number-debugging) or a limit to state in the fact's own sentence, never a number to report with a shrug. Proposing an anchor costs nothing and is reversible — propose it even unasked. Changing the construct's definition to chase the benchmark is the opposite: a metric change, analysis-checkpoints, the user's call.

Show the distribution, not just the center

Economic data is heavy-tailed almost everywhere it matters — income, wealth, firm size, prices, claim amounts. A mean of a heavy-tailed variable can describe no one in the sample, and a mean that moves can move because one tail observation arrived. So:

• Report the median and key percentiles, and the spread (SD, IQR) — not just the center.
• Show the distribution — histogram, density, or ECDF — when the shape carries the point (skew, bimodality, a mass at zero, a cap). The shape is often the fact.
• Prefer a log scale for quantities that vary multiplicatively; a linear axis crushes the bottom 90% against the floor and "the fact" becomes "the top is big."
• A mean that moves with no shape change, and a shape change with no mean movement, are different facts — don't let the mean stand in for both.

Honest visualization

The exhibit should make a real fact legible at a glance, never manufacture one — the chart and the sentence beneath it must say the same thing.

• Axis and scale honesty. Don't truncate a y-axis to exaggerate a wiggle, and don't force a zero baseline onto an index or log scale where it misleads — state the scale you chose. Index to a base year when comparing growth across series of different levels, or the big series reads as "the trend" even when the small one grew faster. Match log vs. linear to the quantity — multiplicative growth is a straight line in logs; forcing it linear hides a constant rate as a late explosion.
• Don't smooth away the variation you're claiming. A loess line through noisy data can invent a trend the scatter doesn't support — show the underlying points or the band.
• Dual-axis traps. Two series on two y-axes can be made to "co-move" by choosing the scales; a co-movement that exists only because of axis choices is not a fact.

Descriptive maps — the same discipline, spatial

A choropleth or point map is a descriptive exhibit, so every rule above has a spatial twin — and maps fail more quietly than charts, because a plausible-looking map is extraordinarily convincing.

• Map a rate, almost never a raw count. A choropleth shaded by the number of events is, to first order, a map of where the population is — the biggest areas light up regardless of the phenomenon. Normalize to a rate or per-capita (the spatial denominator) unless the raw count is genuinely the point.
• The spatial unit is the aggregation level (MAUP) — the same data binned by county, tract, or ZIP can show different, even opposite, patterns. This is the spatial face of the composition choice above: pick the unit deliberately.
• Color binning is the map's version of axis honesty — quantile vs. equal-interval vs. Jenks breaks, and the bin count, change which areas read as extreme. State the scheme; don't hunt for the breaks that dramatize the story.
• The join is where maps silently lie — hand it to data-contracts. A point-to-owner join that fans out double-plots a facility; a point-in-polygon join drops every feature outside all polygons with no error at all. A bad join just looks like a slightly emptier or denser map, so it earns more scrutiny, not less.
• A map shows where, not why — the same causal firewall: "concentrated in X" describes geography, it doesn't claim X caused the concentration.

Describe, don't infer — the causal firewall

The headline guardrail of this arm. A descriptive exhibit shows what co-moves, what differs, what is distributed how — never why — and the verbs have to respect that.

• Descriptive verbs only: "rose alongside," "is higher among," "co-moved with," "is concentrated in," "has widened." Never causal ones: "raised," "drove," "caused," "led to," "increased" (transitive), "the effect of." The verb is the claim.
• A correlation shown is a correlation, full stop — a scatter with a fitted line describes an association, not an effect, no matter how tight the fit.
• The right use of a striking stylized fact is to motivate a question — "Y rose sharply when X was introduced; did X cause it?" routes to causal-identification (or the relevant arm), which earns the causal verb through a design. The descriptive section sets the question up; it does not answer it.
• Descriptive work is legitimate on its own: you don't need a causal design to show a trend, you need one to explain it.

Tooling (R / Python / Julia)

Reach for the simplest row that answers the question. The decomposition row is skipped most and needed most — when a number surprises you, it's the first tool, not the last.

Red flags — STOP

• A dollar trend, count, or rate reported without fixing the comparability choices first — no deflation to a stated base year, a moving denominator, or an unweighted statistic from a weighted sample.
• An aggregate mean/rate/gap that moved or differed across groups with no composition check — within-vs-between never decomposed, subgroups never plotted.
• A mean reported for a heavy-tailed variable with no median, no percentiles, no view of the distribution's shape.
• A stylized fact reported from one specification with no check that it survives an alternative denominator, deflator, weighting, or window.
• A described correlation written with a causal verb ("X raised Y," "the effect of," "drove") — the descriptive-to-causal slide.
• A trend in an average where who is in the sample changed over the window and selection was never ruled out.

Common rationalizations

The Process

1. Name the deliverable — stylized-facts list, summary-stats table, or exploratory pass — so the rigor matches the polish; co-fire question-framing for unit/population/decision if not already pinned.
2. Fix the comparability choices before plotting — denominator, real-vs-nominal + base year, per-capita, weighting, unit, window, aggregation level — annotated with # why:.
3. Run the composition check on anything that moves or differs — within-vs-between (or shift-share/Oaxaca), plot the subgroups, rule out selection-into-sample. The signature step; do it before you believe a surprising number.
4. Show the distribution, not just the center — median + percentiles + shape for anything heavy-tailed; log scale where multiplicative.
5. For a fact you'll report, prove it's robust — re-cut it three reasonable ways; report the robustness as credibility, or downgrade the claim to the range where it holds.
6. Visualize honestly — faithful axes/scales, index for growth comparisons, don't smooth or zoom away the variation you're claiming. For a map, shade a rate not a count, choose the unit and color breaks deliberately, hand the join to data-contracts.
7. Keep the verbs descriptive — "rose alongside," not "raised." Route an emerged question to the fork as motivated; offer result-verification on a fact you're about to ship (the user's call).

The bottom line

Descriptive fact  →  comparability fixed (deflated, per-capita, weighted, one unit) +
                     composition-checked (within-vs-between, subgroups, selection) +
                     robust to alternative cuts + distribution shown + descriptive verbs only
Otherwise         →  a clean-looking artifact of the denominator, the deflator, or the changing mix —
                     and, one causal verb later, a finding the data never supported