Amey-Thakur Schema evolution
A schema shared between producer and consumers is an API. The rules are the same as any API: additive changes flow freely, breaking changes get versions and migration windows, and enforcement beats etiquette.
Method
- Classify every change before making it. Compatible: adding optional/nullable fields with defaults, widening types (int to long), adding enum values consumers treat as open sets. Breaking: removing or renaming fields, retyping, tightening nullability, changing semantics without changing the name (the worst one, because nothing catches it). When unsure, treat it as breaking.
- Enforce compatibility mechanically. A schema registry (Avro/Protobuf/JSON Schema) with backward-compatibility checks in the producer's CI: incompatible publishes fail the build, not the consumers at 3am. For warehouse tables, schema tests on staging models serve the same role (see data-quality-checks).
- Write the contract down with owners. Each shared schema has: owning team, consumers list (or a registry that tracks them), guaranteed fields vs internal ones, and the deprecation process. Data contracts formalize what "you broke our dashboard" arguments litigate after the fact.
- Break with a version and a bridge. New schema version published alongside the old (v2 topic, versioned event type, new table), producer dual-writes or a pipeline translates old-to-new during the migration window, consumers migrate on their schedule with a deadline, usage measured before the old version dies (the api-versioning sunset playbook, applied to data).
- Make consumers evolution-tolerant. Read by field name, ignore
unknown fields, treat enums as open, and never
SELECT *into downstream tables (column order and additions become your problem). Tolerant readers turn most producer changes into non-events. - Propagate deliberately through the stack. A source change flows raw, staging, marts: land new fields in raw automatically, add to staging explicitly, expose in marts as a reviewed change. Lineage (see data-lineage) answers "who is affected"; the impact list turns a schema change from a surprise into a checklist.
Boundaries
- Semantic drift (same field, new meaning: "revenue" now includes refunds) passes every compatibility check and corrupts every consumer; it requires a new field name or a coordinated restatement, never a silent redefinition.
- OLTP migrations inside one service own their tables (see database-migrations); this skill governs schemas that cross team boundaries.
- Registries and contracts do not replace talking to your top consumers before a big change; the tooling enforces the floor, not the courtesy.