PyTorch Experiments
Treat each training run as an experiment you may need to repeat, audit, or resume. Record everything that could change the result, make the data split before any other step, and verify exported models against the original rather than trusting the export.
Step 1: Make the run reproducible
import random, numpy as np, torch
def seed_everything(seed: int) -> None:
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
- Seed each
DataLoader worker with worker_init_fn and a torch.Generator for shuffling. Otherwise workers reuse the same random streams.
- Determinism costs speed. Keep
cudnn.benchmark = False for reproducible comparisons and turn it on only for throughput runs.
- Write a run record next to the checkpoint: git commit, seed, config, library versions (
torch.__version__, CUDA version, GPU name), and dataset hash.
Step 2: Split data before anything else
- Split by the unit that must not leak: patient, session, camera, episode, or sequence. A random frame-level split of correlated frames inflates validation scores.
- Fit normalisation statistics on the training split only.
- Apply augmentation to training data only, after the split.
- Check for leakage explicitly: count identical or near-identical samples across splits and report the number.
Step 3: Training loop essentials
- Save a checkpoint containing model
state_dict, optimiser and scheduler state, the epoch, the best metric, and the RNG state. A checkpoint without optimiser state cannot resume faithfully.
- Use
torch.autocast for mixed precision on CUDA. Confirm loss stays finite, and log any loss-scaling skips.
- Evaluate with
model.eval() and torch.no_grad() (or torch.inference_mode()). Forgetting eval() silently changes dropout and batch-norm behaviour.
- Report the validation metric with the checkpoint it came from, not the last epoch's value.
Step 4: GPU benchmarking and memory
Warm up before timing, synchronise before reading the clock, and report medians over repeated runs:
import time, torch
def benchmark(fn, *, warmup=10, iters=50):
for _ in range(warmup):
fn()
torch.cuda.synchronize()
times = []
for _ in range(iters):
start = time.perf_counter()
fn()
torch.cuda.synchronize()
times.append(time.perf_counter() - start)
times.sort()
return times[len(times) // 2] # median seconds per call
torch.cuda.reset_peak_memory_stats()
# ... run the workload ...
print(torch.cuda.max_memory_allocated() / 2**20, "MiB peak allocated")
nvidia-smi shows reserved memory, which includes PyTorch's cache. max_memory_allocated() shows what tensors actually used.
- If memory is tight, try in this order: smaller batch, gradient accumulation, mixed precision, activation checkpointing (
torch.utils.checkpoint), then a smaller model.
- Compare throughput at the same batch size and precision only.
Step 5: Evaluation and reporting
- Report per-class or per-slice metrics, not only the aggregate. Class imbalance can hide a collapsed class.
- Keep a fixed held-out test set that is touched once, after model selection.
- Store predictions and metrics as files alongside the run record so a result can be re-derived later.
Step 6: ONNX export and verification
import torch
model.eval()
dummy = torch.randn(1, 3, 224, 224, device="cpu") # use the real input shape
torch.onnx.export(
model.cpu(), dummy, "model.onnx",
input_names=["input"], output_names=["output"],
dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
)
Verify the export before using it:
import numpy as np, onnxruntime as ort
session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
ref = model(dummy).detach().numpy()
out = session.run(None, {"input": dummy.numpy()})[0]
np.testing.assert_allclose(ref, out, rtol=1e-3, atol=1e-4)
- Compare on several real inputs, not only the dummy tensor.
- Check the exporter version and opset in the run record. Export APIs change between PyTorch releases.
- For inference speed, benchmark the ONNX model with the same harness as the PyTorch model, on the target hardware.
Reporting
Report the seed, split method and leakage check, the checkpoint used, the metric with its confidence or variance across seeds when available, the hardware, and any step you could not run (for example, the ONNX check on the target device).
1---2name: pytorch-experiments3description: "Run reproducible PyTorch training and evaluation: seeding, deterministic data splits and leakage checks, checkpoints that can resume, GPU memory and throughput benchmarking, and ONNX export with output verification. Use when training or evaluating a PyTorch model, benchmarking a GPU, or preparing a model for deployment."4metadata:5 source: https://skillmd.com/skills/seeramsujay/pytorch-experiments6---78# PyTorch Experiments910Treat each training run as an experiment you may need to repeat, audit, or resume. Record everything that could change the result, make the data split before any other step, and verify exported models against the original rather than trusting the export.1112## Step 1: Make the run reproducible1314```python15import random, numpy as np, torch1617def seed_everything(seed: int) -> None:18 random.seed(seed)19 np.random.seed(seed)20 torch.manual_seed(seed)21 torch.cuda.manual_seed_all(seed)22 torch.backends.cudnn.deterministic = True23 torch.backends.cudnn.benchmark = False24```2526- Seed each `DataLoader` worker with `worker_init_fn` and a `torch.Generator` for shuffling. Otherwise workers reuse the same random streams.27- Determinism costs speed. Keep `cudnn.benchmark = False` for reproducible comparisons and turn it on only for throughput runs.28- Write a run record next to the checkpoint: git commit, seed, config, library versions (`torch.__version__`, CUDA version, GPU name), and dataset hash.2930## Step 2: Split data before anything else3132- Split by the unit that must not leak: patient, session, camera, episode, or sequence. A random frame-level split of correlated frames inflates validation scores.33- Fit normalisation statistics on the training split only.34- Apply augmentation to training data only, after the split.35- Check for leakage explicitly: count identical or near-identical samples across splits and report the number.3637## Step 3: Training loop essentials3839- Save a checkpoint containing model `state_dict`, optimiser and scheduler state, the epoch, the best metric, and the RNG state. A checkpoint without optimiser state cannot resume faithfully.40- Use `torch.autocast` for mixed precision on CUDA. Confirm loss stays finite, and log any loss-scaling skips.41- Evaluate with `model.eval()` and `torch.no_grad()` (or `torch.inference_mode()`). Forgetting `eval()` silently changes dropout and batch-norm behaviour.42- Report the validation metric with the checkpoint it came from, not the last epoch's value.4344## Step 4: GPU benchmarking and memory4546Warm up before timing, synchronise before reading the clock, and report medians over repeated runs:4748```python49import time, torch5051def benchmark(fn, *, warmup=10, iters=50):52 for _ in range(warmup):53 fn()54 torch.cuda.synchronize()55 times = []56 for _ in range(iters):57 start = time.perf_counter()58 fn()59 torch.cuda.synchronize()60 times.append(time.perf_counter() - start)61 times.sort()62 return times[len(times) // 2] # median seconds per call6364torch.cuda.reset_peak_memory_stats()65# ... run the workload ...66print(torch.cuda.max_memory_allocated() / 2**20, "MiB peak allocated")67```6869- `nvidia-smi` shows reserved memory, which includes PyTorch's cache. `max_memory_allocated()` shows what tensors actually used.70- If memory is tight, try in this order: smaller batch, gradient accumulation, mixed precision, activation checkpointing (`torch.utils.checkpoint`), then a smaller model.71- Compare throughput at the same batch size and precision only.7273## Step 5: Evaluation and reporting7475- Report per-class or per-slice metrics, not only the aggregate. Class imbalance can hide a collapsed class.76- Keep a fixed held-out test set that is touched once, after model selection.77- Store predictions and metrics as files alongside the run record so a result can be re-derived later.7879## Step 6: ONNX export and verification8081```python82import torch8384model.eval()85dummy = torch.randn(1, 3, 224, 224, device="cpu") # use the real input shape86torch.onnx.export(87 model.cpu(), dummy, "model.onnx",88 input_names=["input"], output_names=["output"],89 dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},90)91```9293Verify the export before using it:9495```python96import numpy as np, onnxruntime as ort9798session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])99ref = model(dummy).detach().numpy()100out = session.run(None, {"input": dummy.numpy()})[0]101np.testing.assert_allclose(ref, out, rtol=1e-3, atol=1e-4)102```103104- Compare on several real inputs, not only the dummy tensor.105- Check the exporter version and opset in the run record. Export APIs change between PyTorch releases.106- For inference speed, benchmark the ONNX model with the same harness as the PyTorch model, on the target hardware.107108## Reporting109110Report the seed, split method and leakage check, the checkpoint used, the metric with its confidence or variance across seeds when available, the hardware, and any step you could not run (for example, the ONNX check on the target device).