majiayu000 Data modeling
Layers, and why the middle one matters
Three layers, each with one job:
Raw — source data, append-only, otherwise unmodified. Do not apply business logic on ingest: you cannot recover what you discarded, and the logic will need to change retroactively.
Privacy and security transformations are the exception, and belong at ingest. Credentials and secrets should never land in the warehouse at all. Personal data that is not needed should be dropped rather than stored and governed later, and identifiers you must keep but rarely need in the clear should be tokenized or encrypted on arrival. Retention and deletion apply from ingest, not from the marts.
The distinction: strip what you must not hold, keep everything you are entitled to hold, and leave interpretation for later.
Staging — cleaned and conformed: consistent types, standardized names, deduplicated, no business logic yet.
Marts — business-facing models shaped for how questions are asked.
The discipline that pays is keeping business logic out of layers 1 and 2. Logic embedded in ingestion cannot be changed retroactively, and it will need to change.
Grain is the decision everything follows from
State the grain of every table in one sentence: one row per what. "One row per order line per day" is a grain. "Order data" is not.
Most modeling errors are grain errors, and they surface as fan-out — a join multiplying rows so every downstream sum is inflated. If a number is mysteriously too high, check the grain before checking the logic.
Dimensional structure
Facts for events and measurements; dimensions for the things being described. Keep facts narrow and long, dimensions wide and short.
Conform dimensions across facts — one customer dimension, used everywhere. Separate customer tables per domain is how the same customer gets counted differently in two reports.
Handle history deliberately. Overwriting a dimension attribute rewrites the past: last year's revenue silently re-attributes to this year's segment. Decide per attribute whether history matters, and where it does, keep versions with valid-from and valid-to.
The semantic layer
Define metrics once, above the marts, and have every consumer read through it. Without it, the same metric is reimplemented in each dashboard and they drift — not because anyone is careless, but because a filter differs.
The semantic layer is where the metric dictionary becomes executable rather than documentary.
Performance
Model for the query pattern you actually have. Pre-aggregate what is queried constantly; leave the long tail to compute on demand.
Partition and cluster on what people filter by — usually time, then a tenant or entity key. Most slow warehouse queries are full scans of a table that could have been partitioned by date.
Denormalize deliberately, and write down why. Undocumented denormalization is indistinguishable from a modeling error six months later.
Never
- Build a mart directly on raw. The coupling means every source change breaks the business layer.
- Mix grains in one table.
- Let a dashboard contain business logic the warehouse does not. That logic is invisible and unversioned.