Schema Evolution And Compatibility

Schema Evolution And Compatibility

Purpose

Compatibility is a property of a (writer schema, reader schema) pair and a direction — can a reader holding R read bytes written with W? Name the schema/descriptor, parser configuration and application semantics: a JSON validator and a Java DTO binder are separate contracts.

The failure this prevents is the change the registry accepted. The gate runs at registration, per subject, against the configured comparison history; it knows neither which schema a deployed consumer holds nor what is still on the topic — so a change is green on Monday and an outage six months later, when a group resets to earliest.

Workflow

1. Name the pair and the direction: which reader schema, which writer schema, who deploys first.
2. Inventory reachable writers and stored data: deployed and rollback builds, retries, lag, compacted latest values, backups and replay paths. Compaction can retain an old value indefinitely; it does not preserve every historical record. Confirm actual cleanup policies and reachable schema versions before choosing a gate or waiting period (references/runbook-and-ci.md).
3. Look the change up in the format table, carry its condition and not only its verdict, and take the deploy order from the level table — never from the level's name.
4. Confirm the reader actually resolves: specific.avro.reader, the mapper the consumer really uses, the age of its generated code. One that does not voids everything above it.
5. Stage expand and contract, gate the required pairs in CI, and verify migration inventory.

Inspect compiler/toolchain, generated-code and serializer/runtime versions before applying the versioned examples. Transcripts below are historical observations on their stated versions, not validation of the target project. Do not upgrade dependencies just to match them. Return required reader/writer pairs, semantic assertions, rollout/rollback prerequisites and executed versus pending checks. If inventory or decoder configuration is missing, keep the safety verdict conditional.

Compatibility levels: who upgrades first

Named for the new schema, not for who moves: new reader reads old data → reader first.

• Non-transitive checks do not prove compatibility with older versions; they do not prove incompatibility either. For replay, test every reachable writer against the target reader. BACKWARD_TRANSITIVE and FULL_TRANSITIVE can cover this when the compared history is complete; explicit historical-pair tests or verified adapters can also establish it. Preserve schemas, references and identifier resolution needed by retained bytes, including backups.
• Kafka Streams inverts the exception: a Streams app also reads its own changelog and state, which is old-schema data, so Confluent's instruction is to upgrade the Streams apps first and only then the upstream producer for backward-compatible changes. Include state/changelog schemas and restore paths in the matrix; a forward-only gate is insufficient evidence for restoration.

What a change does, by format

Avro rows verified on 1.12.0, Protobuf rows on protobuf-java 4.32.0, Jackson claims on 2.19 and 3.0; JSON Schema is Confluent's STRICT checker on kafka-json-schema-provider 8.3.1.

Expand then contract

Separate compatibility bridges from removal so rollback retains a version that understands both shapes. Additive and subtractive do not universally mean BACKWARD and FORWARD: format, defaults, content model and application behavior decide. This is a replacement-field recipe; verify its pairs.

Release N — EXPAND (additive, readers first)
  1. Add the field: with a default (Avro) / a new number + `optional` (Protobuf) / non-required (JSON).
  2. Register reviewed schemas from CI; select checks covering all required pairs/history.
  3. Deploy CONSUMERS that read both shapes; nothing branches on the new field yet.
  4. Then deploy PRODUCERS that write it, alongside the old field if this is a replacement.
MIGRATE — 5. backfill a mutable store with a race-safe policy for concurrent/old writers;
  6. verify new-field semantics, then prefer it while retaining the old-field fallback.
Release N+1 — CONTRACT (subtractive, writers first)
  7. Deploy PRODUCERS that stop writing the old field.
  8. Prove remaining readers, replay data, retries, restores and rollback builds can use the new
     contract; waiting alone or a zero production rate does not establish this.
  9. Remove obsolete reader dependence and schema fields only when the pair tests permit it
     (Protobuf: reserve number and name; preserve needed historical schemas/Avro aliases).
Rollback must be evaluated as another rollout. Restoring dual writers cannot repair records already
written without the old field; old readers need tested meaningful defaults/adapters or migrated data.

Rules

• record.get("newField") throws AvroRuntimeException: Not a valid schema field although git's newer schema gives it a default. specific.avro.reader defaults to false; the ordinary generic path follows the writer schema unless reader-selection configuration supplies another. Defaults in an unrelated newer schema do not apply. Generic decoding still fails on corrupt data, missing schemas or conversions; successful decoding does not prove application compatibility. Use SpecificRecords, an explicit reader, or deliberate writer-aware generic handling and test the actual configured path.
• Unexpected registrations may bypass schema review. auto.register.schemas defaults to true and overrides use.latest.version and latest.compatibility.strict. Register from CI when schema changes need review, then choose lookup of the application's registered schema or an explicitly intended latest schema. Normalization reduces supported syntactic differences; whitespace alone need not create versions.
• UnrecognizedPropertyException identifies the effective reader policy, not its origin. Bare Jackson 2.x defaults to failing on unknown properties; Jackson 3.0 changes that default. Framework builders may disable it, but custom beans, annotations and explicit configuration matter. Inspect the injected mapper; test production tolerance and separate strict producer/DTO validation so unknown-field tolerance does not hide misspelled required data.
• The Avro union-default rule changed in the 1.12 spec, and vendor documentation still teaches the old one. Under ≤ 1.11 the default must match the first branch, and new Schema.Parser() on 1.11.4 and 1.11.5 reject ["null","string"] with "default":"x"; 1.12.0 says "the first schema that matches" and accepts it. On 1.12.0 and 1.12.1 Schema.Field.defaultVal() returns null for a non-first-branch union default while GenericDatumReader substitutes the value correctly — fixed in the recorded 1.12.2 probe. Verify the deployed version; matching the first branch remains a portable convention (for a nullable field, ["null", "T"] with "default": null).
• A .proto field deleted with a comment instead of reserved is a delayed detonation: reusing the number later at the same wire type yields an exception or a well-formed object with wrong contents, per record, depending on the old payload's bytes. Reserve the number (wire) and the name (ProtoJSON) in the deletion commit.
• IllegalArgumentException: Can't get the number of an unknown enum value. Java closes proto3's open enums with an UNRECOGNIZED constant whose getNumber() and getValueDescriptor() throw, while forNumber returns null and a switch falls to default. Adding a symbol is consumer-first in Java; a reader that must tolerate unknowns reads the generated getXValue() int.
• proto3 presence is a decision, not a detail: an implicit scalar cannot distinguish "cleared" from "the producer's build predates this field" — an explicit 0 puts zero bytes on the wire and hasField() is false. optional has been GA since 3.15.0 ("proto3 removed optional" is 2015 folklore), and unknown-field retention was dropped in 3.0 and restored in 3.5.0, so a proxy that preserves the parsed binary message can retain fields it never heard of. JSON conversion, field-by-field copying or explicit discarding can lose them; test every intermediary.
• An open JSON schema may already allow incompatible values for a newly declared property. Under a backward STRICT gate, adding a restrictive property can fail; an unconstrained property need not narrow the accepted language. Do not edit published v1 to erase the problem. Use a tested bridge/migration or a new contract; LENIENT changes enforcement, not existing data.
• An Avro enum without a default symbol is the cheaper mistake: adding the default in v2 is COMPATIBLE both ways, so the schema edit is free — but every reader deployed on v1 must be replaced before a new symbol can be written. A consumer-first deploy, not a topic rewrite.
• BACKWARD alone leaves a replay evidence gap when old compacted values survive outside the compared history. Check their actual pairs; incompatibility is a possibility, not a consequence of the mode name. kafka-consumers-in-java owns offset resets.

Verification

• Offline, no registry, no network: keep every historical schema in the repo and validate the current one against all of them. On Avro 1.12.0 new SchemaValidatorBuilder().canReadStrategy() with .validateAll() is BACKWARD_TRANSITIVE, .validateLatest() BACKWARD, mutual read FULL. Match the strategy to the registry's level, or CI is greener than production.
• mvn io.confluent:kafka-schema-registry-maven-plugin:8.3.1:test-compatibility against the intended registry checks registered history, as can the compatibility REST API (verbose defaults to true); test-local-compatibility is the offline goal to start CI with.
• buf breaking --against '.git#branch=main,subdir=proto' (buf CLI v1.72.0) on the PR. WIRE_JSON is the floor if anything speaks ProtoJSON — it catches the rename that is free on the wire; FILE if you publish generated Java as a library. It knows nothing of the registry or the topic, so it complements test-compatibility rather than replacing it.
• Test the actual rollout matrix. Serialize with deployed, candidate and retained writer schemas, decode with supported/rollback readers, and assert values, not merely absence of exceptions — that is what catches a changed default (COMPATIBLE from every API, different value) and a same-wire-type reinterpretation, which no metric below can see. For Protobuf use DynamicMessage over two descriptors.
• Signals guide investigation, not proof of completion. Unknown Protobuf field counts can show added/removed fields or wire-type mismatches, including legitimate history. They miss same-wire reinterpretations. Count schema identifiers only after validating framing, excluding tombstones and bounding metric cardinality. Multiple IDs or an unfamiliar ID do not identify a cause. An old ID's observed rate reaching zero does not prove no retained/backup records use it.

References

• Avro — resolution rules verbatim, the verified reader/writer matrix, aliases, the enum default symbol, the union-default divergence across five releases, the fingerprint trap, what a changed default does to old records, SpecificRecord versus GenericRecord. Read when the change is to an .avsc, or when an Avro reader throws.
• Protobuf — the wire-safe, unsafe and lossy lists, presence and generated accessors, the eleven verified type-change outcomes, reused numbers, packed repeated and unknown fields, the UNRECOGNIZED codegen source, oneof, proto2 required. Read when the change is to a .proto.
• Registry and JSON Schema — the Confluent wire format, subject strategies, source-verified serialiser defaults, what the registry does not check, Confluent's JSON Schema rules and content models, Jackson's null-versus-absent gap, Apicurio/Glue/Karapace. Read when the change is to a JSON schema or DTO, or when configuring a registry.
• Runbook and CI — how long "wait" is per store, the options when contraction is impossible, and the CI wiring: Maven plugin configuration, buf.yaml, the Avro validator and round-trip tests, golden bytes, Testcontainers. Read when planning a breaking change's deploy, or when building the gate.