Dvc

DVC (Data Version Control)

Overview

DVC extends Git to the parts Git is bad at: large binary files and reproducible pipelines. Git tracks a small .dvc pointer file (a content hash); DVC stores the actual data in a local cache and, optionally, a remote (S3, GCS, Azure, SSH, or a plain directory). On top of that, DVC provides a lightweight pipeline definition (dvc.yaml) that re-runs only the stages whose inputs changed, and an experiment tracker (dvc exp) that snapshots code+data+params+metrics for each run without needing a tracking server.

Think of it as: Git manages the recipe (code), DVC manages the ingredients (data, models) and can also manage the cooking steps (pipeline stages) and taste-tests (experiments).

DVC is not a replacement for Snakemake/Nextflow when you need complex scheduling, retries, or cluster execution — it complements them. A common pattern is DVC tracking pipeline inputs/outputs while Snakemake or Nextflow drives the actual multi-node execution.

Installation

# Core CLI, local/directory remotes only
uv pip install dvc

# Pick the extra(s) matching your remote storage backend
uv pip install "dvc[s3]"       # AWS S3 (boto3)
uv pip install "dvc[gs]"       # Google Cloud Storage
uv pip install "dvc[azure]"    # Azure Blob Storage
uv pip install "dvc[ssh]"      # SSH/SFTP remote
uv pip install "dvc[all]"      # every backend (heavier install)

# DVCLive for in-training metric logging (separate package)
uv pip install dvclive

Check version: dvc --version (targets 3.67.x; commands below are stable across the 3.x series).

When to Use

• A dataset, model checkpoint, or reference genome is too large for Git (VCF, BAM, FASTQ, .pt/.safetensors weights, indexed references).
• You want make/snakemake-style "only re-run what changed" behavior driven by data hashes, not just file mtimes.
• You need to compare dozens of training runs (params, metrics, code diff) without standing up an MLflow/W&B server.
• You want data lineage that lives in the same Git history as the code that produced it (auditable, PR-reviewable).

Not a fit: real-time experiment dashboards for a team (use W&B/MLflow alongside DVC), or orchestrating thousands of parallel cluster jobs (use Nextflow/Snakemake/Ray, with DVC tracking their inputs/outputs).

Core Concepts

• Cache: content-addressable local store at .dvc/cache (like Git's object store, but for large files).
• .dvc files: small YAML pointers (hash + size + path) that Git tracks in place of the real file.
• Remote: named storage location (S3 bucket, GCS bucket, SSH server, another local dir) DVC pushes/pulls cache objects to/from.
• dvc.yaml: pipeline definition — a DAG of named stages, each with cmd, deps, outs, and optionally params/metrics/plots.
• dvc.lock: auto-generated lockfile recording the exact hashes used in the last successful dvc repro — this is what makes re-runs skippable.
• Experiments: throwaway Git commits (not on your branch) created by dvc exp run, comparable side-by-side with dvc exp show.

Quick Start: Versioning Data

git init
dvc init                      # creates .dvc/, commit this

dvc add data/raw/cohort.vcf.gz     # moves file into cache, writes cohort.vcf.gz.dvc
git add data/raw/cohort.vcf.gz.dvc data/raw/.gitignore
git commit -m "Track cohort VCF with DVC"

# Configure a remote once
dvc remote add -d myremote s3://my-bucket/dvc-store
git add .dvc/config
git commit -m "Add S3 remote"

dvc push        # upload cache objects referenced by tracked .dvc files
dvc pull        # (on another machine/clone) fetch data referenced by .dvc files

dvc add on a directory tracks the whole tree with one .dir hash — good for thousands of small files (e.g., a FASTQ batch) where you don't want thousands of Git-tracked pointers.

Pipelines (dvc.yaml)

Define stages instead of ad hoc dvc add:

# dvc.yaml
stages:
  align:
    cmd: bwa mem ref.fa reads_R1.fq.gz reads_R2.fq.gz | samtools sort -o aligned.bam
    deps:
      - ref.fa
      - reads_R1.fq.gz
      - reads_R2.fq.gz
    outs:
      - aligned.bam

  call_variants:
    cmd: python call_variants.py --bam aligned.bam --out variants.vcf --params params.yaml
    deps:
      - aligned.bam
      - call_variants.py
    params:
      - call.min_qual
    outs:
      - variants.vcf
    metrics:
      - metrics.json:
          cache: false

dvc repro          # runs only stages whose deps/params/code hash changed
dvc dag             # ASCII/graphviz view of the stage DAG
dvc metrics show    # print metrics.json across the tree
dvc metrics diff    # diff metrics vs. the previous commit
dvc params diff     # diff params.yaml vs. the previous commit

params.yaml is a plain YAML file of hyperparameters; dvc.yaml stages reference the specific keys they depend on, so bumping an unrelated param doesn't invalidate every stage.

Experiments

For iterating on a single pipeline without polluting Git history with WIP commits:

dvc exp run -n exp-baseline               # runs the pipeline, snapshots as a hidden Git ref
vi train.py                               # tweak model code
dvc exp run -n exp-more-epochs -S train.epochs=50   # override a param inline

dvc exp show                              # tabular comparison: params, metrics, code
dvc exp diff exp-baseline exp-more-epochs # what changed, numerically
dvc exp apply exp-more-epochs             # promote the winning experiment into your workspace
dvc exp gc                                # clean up experiments you didn't apply

dvc exp run --queue + dvc queue start lets you batch many -S param=value sweeps and run them sequentially or across parallel workers without a scheduler.

DVCLive: In-Training Logging

For frameworks where you want per-epoch metrics without polling files:

from dvclive import Live

with Live() as live:
    for epoch in range(num_epochs):
        train_one_epoch(...)
        live.log_metric("train/loss", loss.item())
        live.log_metric("val/accuracy", val_acc)
        live.next_step()
    live.log_params({"lr": lr, "batch_size": batch_size})

DVCLive writes metrics/params in the format dvc exp show expects, and has drop-in callbacks for PyTorch Lightning, Keras, XGBoost, LightGBM, and Hugging Face Trainer (from dvclive.lightning import DVCLiveLogger, etc.) — no manual log_metric calls needed in those cases.

Common Pitfalls

• Forgetting to dvc push before sharing a repo. A git clone gives collaborators the .dvc pointers, not the data — they must dvc pull (and have remote credentials configured) or dvc pull will error with missing cache objects.
• Committing large files directly with git add instead of dvc add. Once a large blob is in Git history, removing it requires history rewriting; always dvc add first for anything above a few MB.
• Editing a tracked file without going through DVC. If you chmod/edit a file DVC has cached (DVC often uses reflinks/hardlinks or read-only cache files), you can corrupt the cache link. Run dvc unprotect <file> before manual edits, or just dvc add again after editing.
• dvc repro not detecting a change. It hashes deps/params/cmd — if your script reads an undeclared input (e.g., a config file not listed in deps), DVC won't know to re-run. Declare every real input.
• Remote credentials differ per environment. dvc remote modify myremote --local sets machine-specific secrets (keys, tokens) in .dvc/config.local, which is gitignored — don't put credentials in the shared .dvc/config.
• Experiments pile up. dvc exp show gets slow and cluttered after hundreds of runs; periodically dvc exp gc --workspace or push relevant ones and prune the rest.

Resources

• Docs: https://dvc.org/doc
• Command reference: https://dvc.org/doc/command-reference
• DVCLive: https://dvc.org/doc/dvclive
• Source: https://github.com/treeverse/dvc (formerly iterative/dvc)