A result that cannot be regenerated from a command is an anecdote. Reproducibility is cheap at the start and impossible at the end, so this is set up before the first run that will appear in the thesis.
Layout
experiments/
env/ environment.yml or requirements.txt with exact pins; Dockerfile if system deps matter
configs/ one YAML per experiment: exp01-baseline.yaml, exp02-ours.yaml
data/ raw/ (never modified), processed/ (generated by scripts), DATA.md (source, licence, version, checksum)
src/ the code; run.py takes --config and --seed
runs/ one directory per run: <exp>-<seed>-<timestamp>/ with manifest.json, metrics.json, log.txt
results/ aggregated tables (CSV) generated from runs/ by aggregate.py
figures/ generated from results/ by figure scripts (research-figures)
Makefile or justfile: `make exp01` runs all seeds; `make results` aggregates; `make figures`
Everything under runs/, results/, and figures/ is generated; regenerate rather than edit. Commit results/ and figures/ (they are small) so the thesis builds from a clean clone; do not commit runs/ unless small (use DVC or a release asset otherwise).
Environment
- Python:
conda env export --no-buildsorpip freezeintoenv/, with the Python version. Better:pyproject.tomlwith a lockfile (uv lock,poetry lock). - Record the CUDA/driver version and the exact framework build when GPUs are involved; results differ across them.
- A
Dockerfilewhen anything outside the language environment matters (system libraries, a database). - The manifest records
python --version,pip freezehash,nvidia-smioutput, anduname -aat run time regardless.
Determinism
- One
--seedargument that seeds every source:random,numpy,torch(andtorch.cuda.manual_seed_all),tf, data shuffling, and worker init. Use a helperseed_everything(seed)called first. - Set deterministic flags where the framework offers them (
torch.use_deterministic_algorithms(True),CUBLAS_WORKSPACE_CONFIG) and record whether they were on; some ops stay nondeterministic on GPU and the thesis should say so. - Run every experiment with at least three seeds (five or ten when cheap); report mean and SD or CI (
statistics-advisor). A single-seed number is not a result.
Configuration
- All parameters in the config file; nothing hard-coded, nothing from the command line except
--configand--seed(and--out). The config is copied into the run directory unchanged. - Named experiments, not edited-in-place configs:
exp03-ours-lr0.001.yamlrather than changingexp02and losing what produced the earlier result.
The run manifest
runs/<run>/manifest.json, written at start and completed at end:
{
"experiment": "exp02-ours", "seed": 1, "started": "2026-09-21T10:00:00Z", "finished": "...",
"git": { "commit": "a1b2c3d", "dirty": false, "branch": "main" },
"config_sha256": "...", "data_version": "v3 (sha256 ...)",
"env": { "python": "3.12.4", "packages_sha256": "...", "cuda": "12.4", "gpu": "A100 40GB", "host": "..." },
"duration_s": 1832.4, "status": "ok"
}
Refuse to start a run that will appear in the thesis from a dirty working tree (git status --porcelain non-empty): commit first, so the commit hash means something. Log the refusal.
Data
data/DATA.md: source URL, download date, licence, version, checksum of the raw archive, the preprocessing script and its parameters, the resulting row counts and split sizes. Splits are made once with a seed and saved as index files; every run reads the same splits.
Aggregation and figures
aggregate.py reads every runs/*/metrics.json, joins with the manifests, and writes results/<table>.csv with one row per (experiment, seed) and a summary with mean, SD, n. Figure scripts read only results/. The thesis \inputs tables generated from results/ (benchmark-reporting), so a rerun updates the document.
One command
make all (or just all): environment check, all experiments for all seeds, aggregate, figures, tables. Document it in experiments/README.md with the expected total runtime and hardware. The examiner, or a future student, runs it and gets the thesis's numbers.
Checklist before a result enters the thesis
- Run from a clean commit, manifest present, seeds ≥ 3.
- Config committed, data version recorded, environment pinned.
- Table and figure generated from
results/, not typed. - The commit hash cited in the thesis appendix.
