Training Run Provenance

Training Run Provenance

deterministic-core says: no floats in the decision path. In machine learning the floats are the model, so the rule cannot be applied by banning them. It is applied by putting the boundary in the right place:

training run          →   frozen artifact   →   inference   →   decision
nondeterministic          hashed, immutable     deterministic     exact
provenance-sealed         the seam              given the same    integers/decimals,
                                                artifact + input  sealed, auditable

Everything left of the seam is a process you can describe but not always repeat. Everything right of it must be as exact as any other consequential output path in this library. Most ML systems blur the seam — a model reference that means "whatever is at latest", a threshold that drifts, a score compared against a number produced by a different data version — and then the whole pipeline inherits the nondeterminism of the least disciplined stage.

The second thing this skill exists to correct:

A seed is not reproducibility. It makes one program repeat on one machine with one library stack. It says nothing about whether the result holds, and a single-seed number is not a measurement — it is one sample from a distribution nobody characterized.

Composes with the library:

• deterministic-core — everything to the right of the seam
• tamper-evident-audit-chain — the run ledger and the model registry
• data-leakage-hunting — the split must be a stored artifact, which is one of the fields pinned here
• model-evaluation-discipline — variance across seeds is part of the result
• decision-record-discipline — why this architecture, what would reopen it
• honest-degradation — an unreproducible run is reported as such, never rounded up to "reproducible"

Part 1 — The manifest: what a run has to pin

A run that cannot be described cannot be compared, and comparison is the entire business. Capture these at launch, automatically — a manifest written by hand after the fact is a memory, not a record.

Two fields people skip and should not: dirty-tree state (the most common cause of an unreproducible run is uncommitted code) and hardware, because a result that depends on the accelerator model is a real finding and you want to know it now rather than at the next migration.

Then: embed the manifest hash inside the artifact. The question "which run produced the model that made this decision?" must be answerable from the model file alone, months later, with the experiment tracker offline.

Part 2 — Name the nondeterminism you cannot remove

Pretending a training run is deterministic is worse than admitting it is not, because it converts an expected variance into a phantom bug that someone will spend a week chasing. The irreducible sources:

• Float addition is not associative, and parallel reductions sum in whatever order the scheduler produced. Two mathematically identical runs diverge in the last bits, and the divergence compounds across steps.
• Algorithm autotuning picks kernels by measured speed, so the selection depends on machine load.
• Nondeterministic kernels (scatter-add, some pooling and interpolation backward passes) have no deterministic implementation on some backends.
• Dataloader worker interleaving changes batch composition.
• Distributed reduction order varies with topology and timing.
• Library and driver upgrades change numerics with no API change.

What to do, in order:

1. Turn on the framework's deterministic mode and the fixed-algorithm flags where they exist. Measure the throughput cost — it is often 10–30% and it is frequently worth paying for anything you will need to defend.
2. Seed every source, including the dataloader workers and augmentation.
3. Where an op has no deterministic implementation, write it down in the manifest rather than silently accepting it.
4. For anything that must be bitwise reproducible, pin the container image by digest and record the accelerator model, then verify by rerunning and comparing hashes — a claim of bitwise determinism that was never re-run is an assumption.

Part 3 — Label the claim at the rung the evidence supports

The epistemic ladder of red-team-auditing, transplanted. Never let a run float without a level.

UNREPRODUCED with a complete manifest is a perfectly respectable state and describes most training runs ever performed. What is not respectable is calling it reproducible because a seed was set.

The practical consequence for comparisons: two runs that differ by less than the seed-to-seed spread are the same run. Before any "A beats B", you need B's variance. Three seeds is the usual minimum, and reporting the spread costs nothing but two more runs.

Part 4 — The registry is an append-only ledger

Models are evidence. Treat the registry the way tamper-evident-audit-chain treats any record that might be challenged:

• Append-only. New version, never overwrite. A mutable latest tag means the model that produced last Tuesday's decision may no longer exist.
• Immutable references everywhere. Serving config points at a version and a hash, never at a moving tag.
• Each entry seals the previous. Then "which models were deployed between these dates, and in what order" is answerable and verifiable.
• Keep the failed and abandoned runs. They are the variance data, and they are the record of what was already tried — deleting them makes the next person repeat the experiment.
• Log the inference-time inputs to a decision, not just its output (forensic-logging-design), including the model version. Otherwise a disputed decision cannot be reconstructed even with a perfect registry.

And to the right of the seam, deterministic-core applies unmodified: the threshold that converts a score into an action is exact, versioned and sealed; it is not a float that someone tuned in a notebook and typed into a config.

Deliverable

## Run <id> — <purpose>
Data <hash> | Split <hash> | Code <sha, clean|dirty> | Config <hash>
Env: <framework/CUDA/driver/OS> | Accelerator: <model, count>
Seeds: <all of them>
Nondeterminism accepted: <ops/flags that could not be pinned>

Result: <metric> [<interval>] over <n> seeds
Reproducibility: <REPRODUCED-BITWISE | -STATISTICALLY | RERUN-ONCE | UNREPRODUCED>
Artifact: <weights hash>   Manifest hash embedded in artifact: yes/no
Compared against: <run id> — same data version? same split? Δ vs seed spread

Anti-patterns

• "We set seed = 42" offered as a reproducibility claim.
• Storing the split seed instead of the materialized split.
• Reporting a single-seed metric as the result, then comparing it to another single-seed metric and declaring a winner inside the noise.
• Training from a dirty working tree and recording only the commit.
• Pinning Python packages while leaving CUDA, the driver, and the accelerator model unrecorded.
• A deployed model whose training run cannot be identified from the artifact.
• A mutable latest tag in the serving path.
• Deleting failed runs, discarding both the variance data and the record of what was already tried.
• Comparing against a baseline trained on a different data version.
• Letting the nondeterminism of the training run leak past the frozen artifact into the decision, where exactness is still required and still achievable.