Fault Tolerant Training

Fault-tolerant training

On a thousand GPUs for a week, something fails: an uncorrectable ECC error, an NVLink drop, a preempted spot node. A rigid job treats any missing rank as fatal and dies, and a human notices hours later. Fault tolerance turns most failures into a short reconfigure-and-continue instead of a lost run.

Method

1. Run under an elastic launcher. Use torchrun with --max-restarts and a c10d rendezvous, or TorchElastic or Ray Train, so when a worker dies the surviving ranks re-form the process group and resume from the last checkpoint instead of hanging. Set a rendezvous timeout near 600 seconds so a dead node is declared, not waited on forever.
2. Detect hangs, do not block on them. A stuck NCCL collective freezes every rank silently. Set TORCH_NCCL_ASYNC_ERROR_HANDLING=1 and a collective timeout so a stall past the deadline raises instead of deadlocking, and run a heartbeat that kills the job if a rank goes quiet for several minutes.
3. Handle stragglers before they dominate. One GPU thermal-throttling to 70 percent drags every all-reduce to its pace. Log per-rank step time and flag any rank more than 20 percent slower than the median. Cordon that node and let elastic training re-form without it rather than paying the tax each step.
4. Keep spares warm for a fast reconfigure. Hold a few idle nodes in the same rendezvous so a failed rank is replaced in seconds and the world size stays constant. Without spares, dropping ranks shrinks the global batch and quietly shifts the learning-rate schedule.
5. Resume from a verified checkpoint, not the crash point. On restart, load the latest checkpoint that passed its resume check (see checkpointing-large-training), restore dataloader offset and RNG, and reseed so you neither replay nor skip samples. Cap --max-restarts near five so a crash-looping job stops instead of burning GPU-hours.
6. Quarantine bad hardware. After a failure, read the DCGM or Xid error and drain the node from the scheduler pool. A GPU that threw an uncorrectable ECC fault or fell off the bus will fail again; keep a denylist so the launcher does not place ranks back onto it.

Signals

• Does killing one node mid-run trigger an automatic resume within minutes, no one paged?
• Is per-rank step time logged, so you can name the straggler rather than just feel it?
• Does the run cap restarts and denylist hardware, so a flapping node cannot loop forever?
• After a reconfigure, are the global batch size and LR schedule unchanged?

Boundaries

This keeps one run going across failures. It leans on a working checkpoint system for the actual state (checkpointing-large-training) and on the cluster scheduler for node allocation and eviction (gpu-cluster-scheduling). It does not repair the failed hardware or tune the model itself.