Data Versioning Reproducibility

Data Versioning Reproducibility (Verbose)

Core Patterns

Code Versioning Alone Is Not Reproducibility

A git SHA pins the transformation. It says nothing about what went into it. Check out the commit that produced last quarter's model, rerun the pipeline, and the job reads the current tables — with three months of new rows, a corrected billing field, and a column that was backfilled in April. The output is different, and nothing under version control can tell you why.

The claim "our pipeline is reproducible because it's in git" is therefore usually false. Reproducibility of a result requires pinning four things:

Miss any one and the result is not regenerable. In practice teams handle the first and fail the other three.

Content Addressing

Content addressing means the identifier of a dataset is a hash of its bytes. Two useful properties follow. Identical content has one identifier regardless of path or storage system, so deduplication is free. And any mutation changes the identifier, so a silent in-place edit cannot masquerade as the same version.

dvc add data/raw/transactions.parquet
# -> data/raw/transactions.parquet.dvc  (small text file with the content hash)
git add data/raw/transactions.parquet.dvc data/raw/.gitignore
git commit -m "pin transactions snapshot 2026-07"
dvc push                       # bytes go to remote object storage

# On another machine, at any commit:
git checkout <sha> && dvc checkout    # exactly the bytes that commit was built on

Git holds the pointer; object storage holds the bytes. That split is the whole trick — the repository stays small while every commit still names an exact dataset.

The critical distinction, stated plainly: s3://bucket/data/latest.parquet is a variable. md5:8f14e45fceea167a is a value. Pipelines that reference the former have no reproducibility story, no matter how carefully the code is reviewed.

Snapshots for Warehouse Data

Copying a 4 TB table per experiment is not viable. Open table formats solve this with a transaction log: each commit produces a snapshot id, and reading an old snapshot costs nothing extra because the underlying files are still there.

-- Delta Lake
select * from orders version as of 1042;
select * from orders timestamp as of '2026-03-15 00:00:00';

-- Iceberg
select * from orders for system_version as of 3821497;
select * from orders for system_time as of '2026-03-15 00:00:00';

Two operational cautions. First, retention expires history: Delta's VACUUM and Iceberg's snapshot expiration delete the files old snapshots point to, and your seven-day default silently makes six-month-old references unresolvable. Set retention against your actual audit and reproduction window. Second, prefer version as of over timestamp as of when recording provenance — a version id is exact, whereas the timestamp form resolves to whatever snapshot was current then and depends on the same history surviving.

For how the underlying files should be laid out, see data-partitioning; for handling entity history within a table, see slowly-changing-dimensions.

The Run Manifest

Every meaningful run emits one record binding inputs to output. This is the artifact that answers "what produced this?" without archaeology.

import hashlib, json, subprocess
from pathlib import Path

def file_hash(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        for chunk in iter(lambda: fh.read(1 << 20), b""):
            h.update(chunk)
    return f"sha256:{h.hexdigest()}"

manifest = {
    "run_id": run_id,
    "git_sha": subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip(),
    "git_dirty": bool(subprocess.check_output(["git", "status", "--porcelain"])),
    "inputs": {
        "transactions": file_hash("data/raw/transactions.parquet"),
        "orders_snapshot": delta_table.history(1).collect()[0]["version"],
    },
    "params": {"seed": 42, "max_depth": 8, "learning_rate": 0.05},
    "env": {
        "image": "registry/trainer@sha256:2b7c9f...",
        "lockfile": file_hash("uv.lock"),
    },
    "outputs": {"model": "s3://models/churn/v14/model.pkl"},
    "metrics": {"auc": 0.874, "n_train": 1_284_119},
}
Path(f"runs/{run_id}.json").write_text(json.dumps(manifest, indent=2, sort_keys=True))

Record git_dirty. A run from a working tree with uncommitted changes is not reproducible from its SHA, and knowing that at read time is far better than discovering it during an incident.

Pipeline-Level Locking

DVC's pipeline stages give you a make-like dependency graph with content hashes on both sides, so dvc repro re-executes only what actually changed.

# dvc.yaml
stages:
  featurize:
    cmd: python src/featurize.py
    deps:
      - src/featurize.py
      - data/raw/transactions.parquet
    outs:
      - data/processed/features.parquet
  train:
    cmd: python src/train.py --seed 42
    deps:
      - src/train.py
      - data/processed/features.parquet
    params:
      - train.max_depth
      - train.learning_rate
    outs:
      - models/model.pkl
    metrics:
      - metrics.json:
          cache: false

dvc.lock — generated, committed — records the resolved hash of every dependency and output for the last successful run. That file is the lockfile for your data the way uv.lock is for your dependencies, and it is what makes "reproduce commit 4c1e9b2" a mechanical operation rather than a research project.

Determinism

Pinned inputs do not give identical outputs if the process itself is random.

import os, random
import numpy as np
import torch

SEED = 42
os.environ["PYTHONHASHSEED"] = str(SEED)   # must be set before interpreter start
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.cuda.manual_seed_all(SEED)
torch.use_deterministic_algorithms(True)   # raises on non-deterministic kernels
torch.backends.cudnn.benchmark = False     # autotuner picks varying algorithms

Sources of nondeterminism that survive seeding:

• GPU reductions — floating-point addition is not associative, and parallel reduction order varies between runs. Deterministic modes exist and cost 5–30%.
• Thread count — a parallel sum over 8 threads and over 16 threads give slightly different floats. Pin thread counts for bit-exact requirements.
• PYTHONHASHSEED — set inside the process is too late; string hashing is already randomized. Set it in the environment before launch.
• Unordered queries — LIMIT or SAMPLE without a total ORDER BY returns different rows across runs. Any query feeding a hashed artifact needs a deterministic total order.
• Wall-clock in features — now() inside a transformation makes output a function of run time. Pass an explicit as_of parameter instead.
• Dict/set iteration across processes, parallel file listing order, and filesystem-dependent glob ordering.

Decide the standard you need. Bit-exact reproduction is expensive and rarely required. Statistically equivalent — metrics within a stated tolerance — is usually the honest target, and should be written down as one, e.g. "AUC reproduces within ±0.002."

Environment

FROM python:3.11-slim@sha256:9c1f...    # digest, not a tag
COPY uv.lock pyproject.toml ./
RUN uv sync --frozen                    # fail if the lock does not satisfy the spec

A >=1.2 range resolves differently over time, and minor releases of numerical libraries do change results. --frozen (or poetry install --no-update) makes drift a build failure rather than a silent behavior change. Pin the base image by digest: python:3.11-slim is republished and is not a version.

Verify by Actually Reproducing

A reproducibility system that has never been exercised does not work — the failure is always something nobody anticipated, like an expired snapshot or a PYTHONHASHSEED set too late. Schedule the check:

Quarterly: pick a shipped model. From its manifest alone, on a clean machine,
rebuild it. Compare metrics against the recorded values within tolerance.
Record what was missing. Fix that gap.

Common Anti-Patterns

❌ Referencing data by mutable path (s3://bucket/latest/) — the identifier changes meaning without notice. ✅ Content hash or snapshot id, recorded in the run manifest.

❌ Pinning code but not data — "reproducible" is then simply untrue. ✅ Pin code, data, environment, and seeds together.

❌ timestamp as of as the recorded provenance. ✅ Record the exact version/snapshot id; timestamps depend on surviving history.

❌ Default snapshot retention on a table you need to reproduce from. ✅ Set retention to cover your audit and reproduction window explicitly.

❌ Copying datasets into _v2, _v2_final, _v2_final_fixed. ✅ One logical path, versioned by content; the pointer lives in git.

❌ Seeds set but never recorded — you cannot rerun with a seed you did not log. ✅ Seed is a parameter in the manifest, not a constant buried in the script.

❌ Dependency ranges and latest image tags. ✅ Committed lockfile and an image digest.

❌ LIMIT or SAMPLE without a total ORDER BY upstream of a hashed output. ✅ Deterministic total ordering on anything feeding a versioned artifact.

❌ Assuming reproducibility because the tooling is installed. ✅ Periodically rebuild a shipped artifact from its manifest and compare.

Reproducibility Checklist

• Every dataset referenced by content hash or snapshot id, never a mutable path
• Data pointers (.dvc, dvc.lock) committed to git
• Warehouse reads pinned to an explicit table version
• Snapshot retention set to cover the required reproduction window
• Run manifest emitted with code SHA, input hashes, params, env, metrics
• Working-tree-dirty flag captured in the manifest
• All seeds set and recorded as parameters
• Determinism flags enabled where bit-exactness is required
• PYTHONHASHSEED set in the environment, before process start
• No wall-clock calls inside transformations; as_of passed explicitly
• Total ORDER BY on any query feeding a versioned artifact
• Dependency lockfile committed and installs run frozen
• Container base image pinned by digest
• Reproduction tolerance stated explicitly (bit-exact vs metric tolerance)
• A past artifact has been rebuilt from its manifest and verified