Git For Research Code

Git for Research Code

You are an expert in version control for computational optimization research. This skill covers the solo-researcher git workflow: commit granularity tied to experiments, annotated tags that freeze the code state behind every paper table, ignore rules for solver logs and result artifacts, provenance stamping that links each result row to a commit hash, and branch strategies that make risky refactors safe. Use the pattern catalog below to make every reported number reproducible from a single hash.

Initial Assessment

Establish the following before recommending a workflow or writing tooling:

• Repository status. Does a repo exist already? If yes, run git status and git count-objects -vH mentally through the user: is the worktree clean, and has the repo already been polluted with large result files or solver logs?
• Team size. Solo PhD-style work, a 2-3 person lab project, or a larger team? Solo work permits rebase-based history cleanup and direct commits to main; shared remotes require merge discipline and protected branches.
• Artifact inventory. What does a run produce? Typical optimization artifacts: result CSV/parquet tables, solver logs (.log), model files (.lp, .mps), solution files (.sol), checkpoints, convergence traces, figures. Each class needs an explicit track/ignore decision.
• Instance data. Are benchmark instances small text files (TSPLIB-style, fine to commit), large binaries (need LFS or an external store plus checksums), or licensed data that must never enter a public repo?
• Experiment cadence. How often does the user change code and re-run? Daily parameter sweeps need a low-friction stamp-and-run loop; monthly campaign-style runs justify heavier manifests.
• Paper pipeline. Which deliverables consume results - LaTeX tables, figures, a results section? Every camera-ready number should trace back to a tag.
• Reproducibility horizon. Will results need re-running at revision time (6-12 months later)? If yes, environment capture (Python and package versions) must ride along with the git hash.
• Solver licensing. Gurobi license files (gurobi.lic), WLS credentials, and cluster SSH keys must be ignored and scanned for before any push to a public remote.
• Remote and backup. Is there a remote (GitHub/GitLab/institute server)? A repo with no remote is one disk failure away from losing the entire provenance chain.
• Existing damage. If history already contains multi-GB results or secrets, plan a history rewrite (git filter-repo) before adopting the clean workflow, not after.
• Tooling constraints. Python version, OS (path separators in hooks), whether the user can install the pre-commit framework or needs plain .git/hooks scripts.

Workflow Anatomy

The provenance chain

A reported result is reproducible only if it is a pure function of versioned inputs:

$$ \text{result} = \mathrm{Run}(\text{code}@\text{commit},\ \text{config},\ \text{instance},\ \text{seed},\ \text{environment}) $$

Git pins the first argument. The workflow below makes the remaining four ride along with it: configs and instance generators are committed code, seeds and config values are recorded in the result row, and the environment is captured in a per-run manifest. If any argument is missing from the record, the result is an anecdote, not evidence (Sandve et al. (2013), Ten Simple Rules for Reproducible Computational Research).

The chain has four links, each implemented by a pattern in this skill:

commit hash --> run manifest --> result row --> aggregated table --> paper table/figure
   (git)        (JSON sidecar)   (stamped CSV)   (groupby + pivot)     (annotated tag)

What goes in git - artifact taxonomy

Commit granularity - one logical change per commit

The unit of commit is one experiment-relevant change: one operator added, one bug fixed, one parameter default changed, one config introduced. The test: can you write the subject line without the word "and"? Small commits are what make git bisect (find the commit that changed a number), git revert (undo one change), and result-to-commit linking work at all. A 40-file "weekly progress" commit destroys all three.

Commit message convention (subject <= 72 chars, imperative):

  <area>: <what changed>

  <why - the experiment motivation, 1-3 lines>
  <observed effect on results, if known>

Examples:
  ils: add double-bridge kick with strength parameter
  config: raise tabu tenure 8 -> 12 for n>=100 instances
  fix: delta evaluation missed diagonal term (objectives change!)

Flag behavior-changing fixes loudly ("objectives change!") - they invalidate all earlier result rows, and the audit pattern below will show old hashes that must not be mixed with new ones in one table.

Branch vs tag decision

Use annotated tags (git tag -a), never lightweight ones: annotated tags store author, date, and a message describing exactly which tables the snapshot backs, and git describe prefers them.

Provenance Patterns - Linking Results to Code State

Each pattern below is self-contained: motivation, implementation, pitfall. Together they implement the provenance chain end to end.

Pattern 1: Capture the git state at experiment launch

Every experiment script should know which commit it runs from, before it computes anything. Capturing this in-process (rather than by hand) removes the human step that always gets skipped at 2 a.m. before a deadline.

import subprocess
from dataclasses import dataclass


@dataclass(frozen=True)
class GitState:
    """Snapshot of repository state at experiment launch."""
    commit: str     # full 40-char SHA of HEAD
    branch: str     # current branch name, or "DETACHED"
    dirty: bool     # True if tracked files have uncommitted changes
    untracked: int  # count of untracked, non-ignored files


def _git(args: list[str], repo: str = ".") -> str:
    """Run a git command in `repo` and return stripped stdout."""
    out = subprocess.run(
        ["git", "-C", repo, *args],
        capture_output=True, text=True, check=True,
    )
    return out.stdout.strip()


def git_state(repo: str = ".") -> GitState:
    """Capture commit, branch, dirty flag, and untracked count of `repo`."""
    commit = _git(["rev-parse", "HEAD"], repo)
    branch = _git(["rev-parse", "--abbrev-ref", "HEAD"], repo)
    status_lines = _git(["status", "--porcelain"], repo).splitlines()
    untracked = sum(1 for line in status_lines if line.startswith("??"))
    dirty = any(not line.startswith("??") for line in status_lines)
    return GitState(
        commit=commit,
        branch="DETACHED" if branch == "HEAD" else branch,
        dirty=dirty,
        untracked=untracked,
    )


if __name__ == "__main__":
    state = git_state(".")
    print(state.commit[:12], state.branch, "dirty" if state.dirty else "clean")
    # Expected: e.g. "3f2a91c0d4e7 main clean" - short hash, branch, clean flag.

Pitfall: git rev-parse HEAD succeeds even when the worktree is filthy, so the hash alone proves nothing. The dirty flag is the load: a result produced with dirty=True corresponds to no commit at all, because the uncommitted edits are unrecoverable. Also note subprocess with check=True raises CalledProcessError outside a git repo - that is the correct behavior (fail fast), do not swallow it.

Pattern 2: Refuse to launch from a dirty worktree

The cheapest reproducibility guarantee: experiments simply do not start unless the code state is committed. One guard call at the top of every runner script enforces the discipline mechanically.

import subprocess


class DirtyWorktreeError(RuntimeError):
    """Raised when an experiment starts from uncommitted code."""


def _porcelain(repo: str) -> list[str]:
    """Return `git status --porcelain` lines (tracked changes + untracked)."""
    out = subprocess.run(
        ["git", "-C", repo, "status", "--porcelain"],
        capture_output=True, text=True, check=True,
    )
    return out.stdout.strip().splitlines() if out.stdout.strip() else []


def require_clean_worktree(repo: str = ".", allow_untracked: bool = True) -> str:
    """Return HEAD's hash, or raise if tracked files have uncommitted changes.

    Call first in every experiment runner. A result produced from a dirty
    worktree cannot be reproduced from any commit.
    """
    lines = _porcelain(repo)
    blocking = [ln for ln in lines
                if not (allow_untracked and ln.startswith("??"))]
    if blocking:
        details = "\n".join(blocking[:10])
        raise DirtyWorktreeError(
            f"{len(blocking)} uncommitted change(s) in {repo}:\n{details}\n"
            "Commit (or stash) before launching the experiment."
        )
    out = subprocess.run(
        ["git", "-C", repo, "rev-parse", "HEAD"],
        capture_output=True, text=True, check=True,
    )
    return out.stdout.strip()


if __name__ == "__main__":
    commit = require_clean_worktree(".")
    print(f"launching experiment from {commit[:12]}")
    # Expected: prints the short hash on a clean repo; on a dirty repo raises
    # DirtyWorktreeError listing the modified files.

Pitfall: an escape hatch like --allow-dirty gets used once "just to check something quickly," and that one run becomes Table 2. If you must allow exploratory dirty runs, route them to a separate results file (results/scratch_runs.csv) that the paper pipeline never reads. The allow_untracked=True default is deliberate: untracked files are usually new result outputs, not code; tighten it to False when a new source file might silently change behavior (e.g., plugin/factory registration that globs a directory).

Pattern 3: Stamp every result row with commit hash and seed

Result tables outlive memory. A tidy table where every row carries commit, dirty, seed, and parameters answers "which code produced this number?" forever - and lets you filter out rows from superseded code with one boolean mask. Atomic writes (temp file + os.replace) prevent a crashed run from truncating months of accumulated results.

import os
import subprocess
import tempfile
from datetime import datetime, timezone

import pandas as pd


def head_commit(repo: str = ".") -> tuple[str, bool]:
    """Return (HEAD hash, dirty flag) for `repo`."""
    rev = subprocess.run(
        ["git", "-C", repo, "rev-parse", "HEAD"],
        capture_output=True, text=True, check=True,
    ).stdout.strip()
    status = subprocess.run(
        ["git", "-C", repo, "status", "--porcelain"],
        capture_output=True, text=True, check=True,
    ).stdout
    dirty = any(not ln.startswith("??") for ln in status.splitlines())
    return rev, dirty


def stamped_row(instance: str, algorithm: str, seed: int, objective: float,
                runtime_s: float, repo: str = ".") -> dict:
    """Build one tidy result row carrying full code provenance."""
    commit, dirty = head_commit(repo)
    return {
        "timestamp_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "instance": instance,
        "algorithm": algorithm,
        "seed": seed,
        "objective": objective,
        "runtime_s": round(runtime_s, 3),
        "commit": commit,
        "dirty": dirty,
    }


def append_rows_atomic(rows: list[dict], csv_path: str) -> None:
    """Append rows to a CSV using write-to-temp + os.replace (crash-safe)."""
    new = pd.DataFrame(rows)
    combined = (pd.concat([pd.read_csv(csv_path), new], ignore_index=True)
                if os.path.exists(csv_path) else new)
    directory = os.path.dirname(os.path.abspath(csv_path))
    os.makedirs(directory, exist_ok=True)
    fd, tmp = tempfile.mkstemp(dir=directory, suffix=".csv.tmp")
    os.close(fd)
    combined.to_csv(tmp, index=False)
    os.replace(tmp, csv_path)


if __name__ == "__main__":
    rows = [stamped_row("rand_n50_s0", "tabu", seed=s,
                        objective=1234.5 - s, runtime_s=2.0)
            for s in range(3)]
    append_rows_atomic(rows, "results/raw_runs.csv")
    table = pd.read_csv("results/raw_runs.csv")
    print(table[["seed", "objective", "dirty"]].to_string(index=False))
    # Expected: 3 rows, all sharing one 40-char commit hash and dirty=False.

Pitfall: stamping the aggregated table instead of the raw runs. Aggregations (means over seeds, best-of-10) often mix rows from several commits without anyone noticing; only per-run stamping makes the mixing detectable. Keep one row per run in the raw file and rebuild aggregates from it (see pandas-experiment-management for the tidy-table discipline). Also: read-concat-rewrite is O(file size) per append - fine up to ~10^5 rows; beyond that, write per-run files and concatenate at analysis time.

Pattern 4: Audit a result table's provenance before using it

Before a table feeds a paper, verify two invariants: every referenced commit still exists in the repo (it can vanish after a rebase or an aggressive gc of unreachable objects), and no dirty rows are present.

import subprocess

import pandas as pd


def commit_exists(commit: str, repo: str = ".") -> bool:
    """True if `commit` resolves to a commit object in `repo`."""
    res = subprocess.run(
        ["git", "-C", repo, "cat-file", "-e", f"{commit}^{{commit}}"],
        capture_output=True, text=True,
    )
    return res.returncode == 0


def audit_provenance(csv_path: str, repo: str = ".") -> pd.DataFrame:
    """Per-commit report over a stamped result table: rows, dirty rows, existence."""
    df = pd.read_csv(csv_path)
    report = (
        df.groupby("commit")
          .agg(rows=("commit", "size"), dirty_rows=("dirty", "sum"))
          .reset_index()
    )
    report["in_repo"] = report["commit"].map(lambda c: commit_exists(str(c), repo))
    return report.sort_values("rows", ascending=False, ignore_index=True)


if __name__ == "__main__":
    report = audit_provenance("results/raw_runs.csv")
    print(report.to_string(index=False))
    bad = report[(~report["in_repo"]) | (report["dirty_rows"] > 0)]
    print("provenance problems:", len(bad))
    # Expected: "provenance problems: 0" when every hash resolves and no row
    # was produced from a dirty worktree.

Pitfall: rebasing or amending commits that already stamped result rows orphans those hashes - commit_exists returns False even though "the same code" exists under a new hash. Rule: history rewriting is allowed only for commits that no result row references yet. If you must clean history before publishing the repo, do it after tagging and re-running the final tables, or keep the old hashes reachable via a archive/pre-cleanup branch.

Snapshot Patterns - Tags for Paper Results

Pattern 5: One annotated tag per paper artifact

When a table or figure is final, freeze the exact code state behind it. Annotated tags are the right object: immutable, named, carrying a message that says which artifact they back. The naming scheme makes them sortable and greppable.

Tag naming scheme:

  paper-<venue><year>-<round>-<artifact>

  paper-ejor2026-r0-table3      code behind Table 3, initial submission
  paper-ejor2026-r0-fig4        code behind Figure 4 convergence plot
  paper-ejor2026-r1-table3      Table 3 after first revision
  thesis-ch4-final              chapter-level snapshot

One tag per artifact (not one per paper) when tables were produced at
different commits; one tag for everything when the whole results section
was regenerated from a single commit (the cleaner setup - aim for it).

import subprocess
from pathlib import Path


def tag_paper_snapshot(tag: str, message: str, repo: str = ".") -> None:
    """Create an annotated tag freezing the code state behind a paper artifact."""
    subprocess.run(
        ["git", "-C", repo, "tag", "-a", tag, "-m", message],
        check=True,
    )


def export_snapshot(tag: str, out_dir: str = "snapshots", repo: str = ".") -> Path:
    """Export the tagged tree as a zip (no .git history) for archival upload."""
    out = Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)
    target = (out / f"{tag}.zip").resolve()
    subprocess.run(
        ["git", "-C", repo, "archive", "--format=zip",
         f"--output={target}", tag],
        check=True,
    )
    return target


if __name__ == "__main__":
    tag_paper_snapshot(
        "paper-ejor2026-r0-table3",
        "Code state for Table 3: ILS vs SA, 30 instances x 10 seeds, "
        "configs/campaign_table3.json, results in raw_runs.csv commit-filtered.",
    )
    path = export_snapshot("paper-ejor2026-r0-table3")
    print("archived:", path)
    # Expected: annotated tag created; snapshots/paper-ejor2026-r0-table3.zip
    # contains the exact tagged tree, suitable for Zenodo/journal upload.

Pitfall: tags are local until you git push --tags (or git push origin <tag>); a disk failure before pushing destroys the snapshot. Push tags immediately after creating them. Second trap: re-pointing an existing tag (git tag -f) after the paper is submitted silently breaks the paper-to-code link - if the code must change, create a new tag for the new round instead.

Pattern 6: Re-run an old result without disturbing current work

Reviewers ask for reruns months later. Checking out an old commit in your main worktree stalls current development and risks mixing states. git worktree gives you the old code in a separate directory while main keeps moving.

import subprocess
from pathlib import Path


def add_rerun_worktree(commit: str, repo: str = ".",
                       root: str = "../reruns") -> Path:
    """Check out `commit` into a fresh detached worktree for re-running.

    The main checkout stays untouched; current development continues in
    parallel with the rerun.
    """
    dest = Path(root) / commit[:12]
    subprocess.run(
        ["git", "-C", repo, "worktree", "add", "--detach", str(dest), commit],
        check=True,
    )
    return dest


def remove_rerun_worktree(dest: Path, repo: str = ".") -> None:
    """Remove a finished rerun worktree and prune git's bookkeeping."""
    subprocess.run(
        ["git", "-C", repo, "worktree", "remove", "--force", str(dest)],
        check=True,
    )
    subprocess.run(["git", "-C", repo, "worktree", "prune"], check=True)


if __name__ == "__main__":
    target = "paper-ejor2026-r0-table3"   # tags and raw hashes both work
    dest = add_rerun_worktree(target)
    print("re-run inside:", dest)
    # Expected: ../reruns/<short-ref>/ holds the exact old tree; run the
    # recorded config + seed there and compare objectives row by row.

Pitfall: the worktree pins the code, not the environment. If numpy or Gurobi changed versions since the original run, objectives can differ (different default parameters, different tie-breaking, different RNG streams across major numpy versions). That is why Pattern 7's manifest records package versions: recreate the environment (pip install numpy==1.26.4 ... or a lockfile) inside a fresh virtualenv before re-running. Note add_rerun_worktree(commit[:12]) builds dest from the first 12 chars of whatever ref you pass - for tag names the directory is just the truncated tag, which is fine.

Pattern 7: Per-run manifest - commit + config + environment

The result row holds the scalars; the manifest holds everything else needed to re-run: full config, interpreter version, OS, and the exact versions of the packages that influence numerics.

import json
import platform
import subprocess
import sys
from datetime import datetime, timezone
from importlib import metadata
from pathlib import Path


def environment_snapshot(packages: list[str]) -> dict:
    """Record interpreter, OS, and installed versions of numerics packages."""
    versions: dict[str, str] = {}
    for pkg in packages:
        try:
            versions[pkg] = metadata.version(pkg)
        except metadata.PackageNotFoundError:
            versions[pkg] = "not installed"
    return {
        "python": sys.version.split()[0],
        "platform": platform.platform(),
        "packages": versions,
    }


def write_run_manifest(run_id: str, config: dict, repo: str = ".",
                       out_dir: str = "results/manifests") -> Path:
    """Write a JSON sidecar binding one run to commit, config, and environment."""
    commit = subprocess.run(
        ["git", "-C", repo, "rev-parse", "HEAD"],
        capture_output=True, text=True, check=True,
    ).stdout.strip()
    manifest = {
        "run_id": run_id,
        "timestamp_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "commit": commit,
        "config": config,
        "environment": environment_snapshot(["numpy", "pandas", "gurobipy"]),
    }
    out = Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)
    path = out / f"{run_id}.json"
    path.write_text(json.dumps(manifest, indent=2), encoding="utf-8")
    return path


if __name__ == "__main__":
    cfg = {"algorithm": "ils", "max_iter": 20000, "kick_strength": 4, "seed": 7}
    path = write_run_manifest("ils_rand_n50_seed7", cfg)
    print("manifest:", path)
    # Expected: results/manifests/ils_rand_n50_seed7.json containing the commit
    # hash, the full config dict, and pinned numpy/pandas/gurobipy versions.

Pitfall: dumping pip freeze (hundreds of lines) into every manifest creates noise that hides the signal; record only packages whose version can change numerics (solver, numpy/scipy, your own package). Conversely, recording no environment is the classic failure: "same commit, different numbers" at revision time is almost always a solver or numpy upgrade, and without the manifest you cannot prove it.

Hygiene Patterns - Ignore Rules, Hooks, and Repository Size

Pattern 8: A .gitignore built for optimization work

Solver and experiment artifacts appear in predictable shapes. Ignore them from day one - retro-fitting after results/ is in history requires a rewrite.

# --- experiment outputs (regenerate from commit + config + seed) ---
results/
snapshots/
figures/
*.csv
*.parquet
!data/instances/**/*.csv        # committed benchmark instances stay tracked

# --- solver artifacts ---
*.log                            # Gurobi/CP-SAT/HiGHS logs
*.lp
*.mps
*.sol
*.ilp                            # IIS output
*.prm                            # parameter dumps
*.rew *.dua *.bas

# --- secrets and licenses: NEVER in a repo ---
gurobi.lic
*.lic
.env

# --- python noise ---
__pycache__/
*.pyc
.venv/
*.egg-info/

# --- editor/OS noise ---
.idea/ .vscode/
.DS_Store Thumbs.db

Pitfall: the blanket *.csv rule also hides input data; the negation line (!data/instances/...) re-includes committed benchmarks, but git cannot re-include a file inside an ignored directory - which is why the rule ignores *.csv by extension and results/ by directory, never data/ itself. Test the net effect with git check-ignore -v <path> whenever a file mysteriously refuses to be added.

Pattern 9: Pre-commit guard against artifacts and oversized files

.gitignore does not stop git add -f or a pattern gap. A pre-commit hook is the hard boundary: commits containing result artifacts or large blobs fail with an explanation.

import subprocess
import sys

MAX_BYTES = 5_000_000
ALLOWED_PREFIXES = ("data/instances/",)
BLOCKED_PREFIXES = ("results/", "logs/", "snapshots/", "figures/")
BLOCKED_SUFFIXES = (".log", ".sol", ".lp", ".mps", ".ilp", ".parquet")


def staged_files() -> list[str]:
    """Paths of files added/copied/modified in the staged commit."""
    out = subprocess.run(
        ["git", "diff", "--cached", "--name-only", "--diff-filter=ACM"],
        capture_output=True, text=True, check=True,
    )
    return out.stdout.split()


def staged_blob_size(path: str) -> int:
    """Size in bytes of the staged version of `path` (index, not worktree)."""
    out = subprocess.run(
        ["git", "cat-file", "-s", f":{path}"],
        capture_output=True, text=True, check=True,
    )
    return int(out.stdout.strip())


def main() -> int:
    """Block commits that include experiment artifacts or oversized blobs."""
    failures: list[str] = []
    for path in staged_files():
        if path.startswith(ALLOWED_PREFIXES):
            continue
        if path.startswith(BLOCKED_PREFIXES) or path.endswith(BLOCKED_SUFFIXES):
            failures.append(f"{path}: experiment artifact - keep out of git")
        elif staged_blob_size(path) > MAX_BYTES:
            failures.append(f"{path}: {staged_blob_size(path)} B > {MAX_BYTES} B")
    for line in failures:
        print(f"pre-commit: {line}", file=sys.stderr)
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())
    # Expected: exit 0 on a clean commit; exit 1 listing each blocked file,
    # which aborts the commit.

Installation (plain hook, no framework needed):

  cp scripts/precommit_guard.py .git/hooks/   # keep source in scripts/ too
  printf '#!/bin/sh\nexec python scripts/precommit_guard.py\n' > .git/hooks/pre-commit
  chmod +x .git/hooks/pre-commit

Or with the pre-commit framework (.pre-commit-config.yaml):

  repos:
    - repo: local
      hooks:
        - id: artifact-guard
          name: block experiment artifacts and large files
          entry: python scripts/precommit_guard.py
          language: system
          pass_filenames: false

Pitfall: hooks live in .git/hooks/, which is not versioned - a fresh clone has no hooks. Keep the hook script in scripts/ (versioned) and document the one-line install, or use the pre-commit framework whose config file is versioned. Second trap: checking worktree file size instead of staged blob size (git cat-file -s :<path>) misses the case where a huge file was staged and then truncated on disk.

Pattern 10: Regression gate for risky refactors

A refactor branch earns its merge by proving behavior is unchanged: same commit-pinned configs and seeds must give identical objectives. Run the baseline suite on main, run it on the branch, and diff per-(instance, algorithm, seed).

import pandas as pd


def compare_baselines(old_csv: str, new_csv: str,
                      tol: float = 1e-9) -> pd.DataFrame:
    """Join per-(instance, algorithm, seed) objectives from two result tables.

    For a behavior-preserving refactor with fixed seeds, every delta must be
    zero (up to floating-point reordering tolerance).
    """
    keys = ["instance", "algorithm", "seed"]
    old = pd.read_csv(old_csv)[keys + ["objective"]].rename(
        columns={"objective": "objective_old"})
    new = pd.read_csv(new_csv)[keys + ["objective"]].rename(
        columns={"objective": "objective_new"})
    merged = old.merge(new, on=keys, how="outer", indicator=True)
    merged["delta"] = merged["objective_new"] - merged["objective_old"]
    merged["regression"] = ((merged["_merge"] != "both")
                            | (merged["delta"].abs() > tol))
    return merged


if __name__ == "__main__":
    report = compare_baselines("results/baseline_main.csv",
                               "results/baseline_refactor.csv")
    bad = report[report["regression"]]
    print(f"{len(bad)} regressions out of {len(report)} paired runs")
    if len(bad) > 0:
        print(bad[["instance", "algorithm", "seed", "delta"]].head(10)
              .to_string(index=False))
    # Expected: "0 regressions out of N paired runs" for a pure refactor;
    # any nonzero delta means the branch changed behavior, not just structure.

Pitfall: runtime is not part of the equality gate - a refactor for speed should change runtime_s and nothing else. But beware floating-point summation order: vectorizing a loop can change objectives in the 12th digit legitimately. Set tol to the level your objective arithmetic guarantees (exact integers: tol=0; float accumulation: 1e-9 relative is typical), and document the choice next to the gate.

Advanced Techniques

Bisecting a results regression

When a probe instance's objective silently changed somewhere in the last 60 commits, git bisect run finds the culprit commit in O(log n) probe runs - this is the payoff of small commits. Write a check script that exits 0 (good), 1 (bad), or 125 (skip: commit cannot run the probe).

"""Bisect probe: exit 0 if the solver still reproduces the reference objective.

Usage:
    git bisect start <bad-commit> <good-commit>
    git bisect run python scripts/bisect_check.py
"""
import subprocess
import sys

REFERENCE_OBJECTIVE = 1234.5
TOLERANCE = 1e-9


def run_probe() -> float:
    """Run the fast probe experiment and parse the printed objective."""
    out = subprocess.run(
        [sys.executable, "scripts/run_experiment.py",
         "--config", "configs/probe_small.json", "--seed", "7"],
        capture_output=True, text=True, timeout=120,
    )
    if out.returncode != 0:
        raise RuntimeError(out.stderr[-500:])
    last_line = out.stdout.strip().splitlines()[-1]   # e.g. "objective=1234.5"
    return float(last_line.split("=")[1])


def main() -> int:
    """Map the probe outcome onto git-bisect exit codes (0 good, 1 bad, 125 skip)."""
    try:
        objective = run_probe()
    except Exception:
        return 125    # this commit cannot run the probe: tell bisect to skip it
    return 0 if abs(objective - REFERENCE_OBJECTIVE) <= TOLERANCE else 1


if __name__ == "__main__":
    sys.exit(main())
    # Expected: `git bisect run` converges to the first behavior-changing
    # commit in ~log2(#commits) probe executions, then `git bisect reset`.

Keep the probe under ~2 minutes (small instance, short iteration budget, fixed seed); bisect may run it 8-12 times. The probe config must exist across the whole bisect range - if you renamed configs mid-range, have the script fall back or return 125.

Branch strategy for risky refactors

Protocol: (1) on main, run the baseline suite and commit/record its results file with the commit hash; (2) branch refactor/<topic>; (3) commit in small steps, running the probe instance after each step so bisect stays usable within the branch; (4) at the end, run the full baseline suite and apply Pattern 10's gate; (5) merge with --no-ff so the branch shape survives in history, or rebase-then-merge if you prefer linear history and no result rows reference the branch commits yet (Pattern 4's pitfall). Abandoned branches: keep them under try/ for a month, then delete - the reflog and the experiment log entry preserve what was learned.

Paper repository coupling

Two workable setups. (a) Paper inside the code repo (paper/ directory): a single tag freezes code and text together; ignore LaTeX build artifacts (*.aux, *.bbl, *.synctex.gz). (b) Separate paper repo: store the code tag name and commit hash in a provenance.tex comment block next to each table, and consider a git submodule pin if reviewers get repo access. In both setups, generate LaTeX tables programmatically from the stamped results file rather than pasting numbers - the generating script plus the commit hash is the audit trail (see pandas-experiment-management for DataFrame.to_latex pipelines).

Large instances and external data

Committing 500 MB of benchmark instances makes every clone painful. Options, in order of preference: (1) commit the generator + seeds and regenerate (best - generators are tiny and exact); (2) a versioned download script plus a committed checksums.sha256 file, verified before experiments run; (3) git lfs track "data/large/**" when the data is irreplaceable and must live with the repo (mind LFS bandwidth quotas on public hosts); (4) DVC when datasets themselves evolve and need their own versioning. Never option zero - raw blobs in plain git history.

Cleaning history before publishing the repo

Before making a thesis/paper repo public: scan for secrets (gurobi.lic, tokens) over all history, not just HEAD - git log --all --diff-filter=A --name-only or a scanner like gitleaks; strip accidentally committed bulk with git filter-repo --path results/ --invert-paths (operate on a fresh clone; the rewrite changes all downstream hashes). Because rewriting breaks the hash links in stamped result tables, do this only after final tables are regenerated, or publish a clean export (Pattern 5's git archive) instead of the living repo.

Practical Challenges

Old results no longer reproduce after months of development. The result rows carry commit hashes (Pattern 3) - use Pattern 6 to open a worktree at the recorded hash, recreate the environment from the run manifest (Pattern 7), and re-run with the recorded config and seed. If they still differ, the run was under-specified: typically an unseeded RNG, wall-clock-dependent stopping, or thread-count-dependent solver behavior (fix Threads and Seed in Gurobi params).

The repo is 2 GB because results and logs were committed early on. Stop the bleeding first: add the ignore rules (Pattern 8) and git rm -r --cached results/ so new commits are clean. Then schedule a history rewrite with git filter-repo on a fresh clone - but only after auditing which stamped hashes would be orphaned (Pattern 4) and re-tagging final states.

Forty commits in a row named "fix" or "wip". History like this defeats bisect and code archaeology. Going forward, adopt the subject-line convention (<area>: <change>) and commit per logical change. For the existing mess: do not rewrite if result rows reference those hashes; instead, write one annotated tag now with a message summarizing the state, and start the discipline from here.

A reviewer asks for new experiments 9 months after submission. Check out the paper tag in a worktree, rebuild the environment from the manifests, re-run the baseline suite, and verify it matches the submitted table before running anything new. If it matches, run the new experiments from the same tag (or from a revision-r1 branch off the tag if code changes are needed) and tag the new state paper-<venue>-r1.

Results were produced from a dirty worktree and it is in the paper. Quantify the damage: dirty rows are flagged in the table (Pattern 3). Re-run those exact (instance, algorithm, seed) cells from the nearest clean commit; if the numbers match, replace the rows and note nothing; if they differ, the published numbers are unreproducible and must be regenerated wholesale before camera-ready. Then install Pattern 2 so it cannot recur.

A refactor silently changed objective values. If the regression gate (Pattern 10) was skipped, bisect (Advanced Techniques) between the last-good and first-bad commits with a fast probe. The most common culprits in optimization code: changed iteration order altering RNG consumption, delta-evaluation bugs, and floating-point reassociation from vectorization.

Binary or licensed instance data bloats or legally contaminates the repo. Keep licensed data out entirely: a download script with checksum verification, plus an ignore rule for data/licensed/. For bulky open data, prefer regenerating from committed generators with seeds; the generator is the ground truth and diffs are reviewable.

Experiments run on a cluster, development happens on a laptop. Push before submitting jobs; the job script's first action is git fetch && git checkout <hash> into a job-specific worktree or clone, with the hash passed explicitly by the submit script - never "whatever the shared checkout currently has." Stamp rows with hostname as well; cross-machine numeric drift (different BLAS, different CPU) then becomes diagnosable.

The Gurobi license file got committed and pushed. Treat it as leaked: remove it from all history (git filter-repo --path gurobi.lic --invert-paths), force-push, and rotate the license/WLS credentials with the vendor. Add *.lic to both the repo .gitignore and your global ignore file (git config --global core.excludesFile).

Two tables in the paper turn out to come from different commits. Pattern 4's audit shows the mix. Decide which commit is canonical, re-run the stale table's cells from it, and from then on regenerate the entire results section from one commit per revision round - one tag, one hash, every table.

Tools & Libraries

Output Format

A complete answer on this topic delivers checklists and templates the user can adopt verbatim.

REPOSITORY SETUP CHECKLIST (once per project)
[ ] git init; add remote; first push
[ ] .gitignore from Pattern 8 (results, solver logs, secrets, python noise)
[ ] pre-commit guard installed and its source committed under scripts/
[ ] data/ policy decided: generators+seeds | download+checksums | LFS
[ ] experiment runner calls require_clean_worktree() and stamps rows
[ ] manifests directory + environment_snapshot wired into the runner
[ ] global ignore for *.lic and editor noise (core.excludesFile)

PER-EXPERIMENT LOOP CHECKLIST (every iteration)
[ ] one logical change -> one commit, subject "<area>: <change>"
[ ] beha

…(truncated)