Reproducible Training Runs

Purpose

Enforce seed setting, deterministic operations, and environment tracking so a training run can be reproduced exactly.

When to Use

• Reviewing or modifying a training script that must be deterministic
• Two runs with the "same" configuration produced different results
• A past result needs to be reconstructed exactly

Inputs

• A target Python training script

Workflow

1. Global seed initialization: ensure a single function sets seeds for all relevant libraries (random, numpy, torch, tensorflow).
2. Deterministic algorithms: for PyTorch or TensorFlow, check that deterministic algorithms are enabled (e.g., torch.use_deterministic_algorithms(True)).
3. Data loading: verify data loaders use deterministic shuffling and that worker processes are seeded correctly to avoid identical augmentations.
4. Environment & config tracking: ensure the script logs the exact configuration, dependency versions, and data hashes.
5. Review first: point out missing reproducibility guards before rewriting the script. Provide the exact seed-initialization snippet — do not hide side effects.

Output

• A list of missing reproducibility guards with the exact code snippets to add, plus any performance trade-off warnings

Verification

• All library seeds set from one place
• Deterministic-algorithm flags enabled (or the gap explicitly accepted)
• Loader shuffling and worker seeding deterministic
• Configuration, dependency versions, and data hashes logged
• User warned if determinism flags significantly slow training

Failure Modes

• Partial seeding — seeding random but not the framework or loader workers still yields nondeterminism.
• Silent slowdown — enabling deterministic algorithms can cost real training speed; surface the trade-off instead of hiding it.
• Rewriting before reviewing — changing the script without first listing the gaps loses the audit trail.