annatchijova Versioned Schema Evolution
Any format you persist will outlive the code that wrote it. The moment a saved file, a database row, or a serialized model can be loaded by a different version of your code than created it, you have a schema-evolution problem — and if the format carries no version, you have an unsolvable one: you cannot safely change it, because you cannot tell what you are looking at. The version stamp is not metadata; it is the contract that makes the format evolvable at all.
Stamp the version — in the artifact, not just the code
Write a version field into every serialized artifact ("engine_version": "pca_v2.4"). This is distinct from a hashing version (see deterministic-core, which versions the canonical form so a seal stays stable): this version governs deserialization — it tells the loader which shape it is about to read. Increment it whenever the serialized shape changes in a way that affects how it must be read back.
Detect the version on load, and decide deliberately
A loader must read the stamped version first and branch on it. There are three honest responses to a version it sees:
- Known and current — load directly.
- Known but older — migrate forward (below), or load with documented best-effort compatibility.
- Unknown / newer than this code — warn loudly and proceed best-effort, or refuse, depending on how load-bearing the data is. Never load a future version silently as if it were the current one; the fields you assume may not mean what you think.
A version mismatch is information the caller needs, surfaced as a warning, not swallowed.
Migrate old shapes forward
When changes accumulate, a chain of small migrators beats a pile of conditionals. Register a migrator per version step (v1 -> v2, v2 -> v3) and, on load, apply them in sequence from the artifact's version up to the current one. Each migrator does one thing: add the field, rename the key, split the column. The loader never special-cases "what if this is v1 being read by v3 code" — it just walks the chain. This keeps every individual migration small and reviewable, and it means old data is always readable by new code.
Backward compatibility: a missing field is not a zero
The subtle trap is filling a field that an old schema didn't have. If v1 saved no mean_vector and v2 needs one, defaulting it to zeros produces an object that loads cleanly and is wrong — the model is now centered on the wrong point and every result over it is quietly off. The honest move is to fill the safe default and mark the artifact as degraded (requires_rebuild = True) so a staleness check rebuilds it from real data instead of trusting the reconstruction. That hand-off to honest-degradation is the difference between "backward compatible" and "backward compatible and silently incorrect".
Match the strictness to the source
Deserialization strictness depends on what the source can preserve. JSON cannot encode a dtype — every number comes back as a 64-bit float — so a JSON loader must coerce (with a warning), not reject, on dtype. A binary source (pickle, .npy) does preserve dtype, so there a mismatch is a real corruption and should raise. A single strict flag, defaulted correctly per source, expresses this: strict=False for text round-trips, strict=True for binary.
Keep the old readers alive
When you bump the version, do not delete the ability to verify or read the previous one. Historical artifacts — sealed bundles, audit records, archived models — were written under the old schema and must still validate. "Increment the version and keep supporting verification for the prior version" is the rule; a migration that orphans everything written before it is a data-loss event wearing a refactor's clothes.
Tooling
scripts/schema_versioning.py is a small, dependency-free framework: dump() stamps the current version, load() detects the stored version, walks a registered chain of migrators up to current, warns on unknown/newer versions, and supports per-source strictness. It includes a worked v1 -> v2 migration. Run it to see an old-shape record loaded, migrated, and flagged.