# Exact Training Resume Guard

> Enforce exact-resume support for long-running training jobs. Use when writing or updating any training script, launcher, sbatch file, DeepSpeed/Accelerate/TRL training entrypoint, or checkpoint policy where future runs must resume from the last step with optimizer, scheduler, RNG, and framework state preserved after timeout, preemption, or manual interruption.

- Skill: `kwongfuk/exact-training-resume-guard` (Agent Skill)
- Install (CLI): `npx skillmds@latest add kwongfuk/exact-training-resume-guard`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kwongfuk/exact-training-resume-guard/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: KwongFuk (https://skillmd.com/u/kwongfuk)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/kwongfuk/exact-training-resume-guard

---


# Exact Training Resume Guard

Use this skill whenever a task writes or edits training launchers, training entrypoints, or checkpoint logic.

This is not "load model weights and start over." The requirement is exact resume:
- continue from the last saved step
- preserve optimizer state
- preserve scheduler position
- preserve RNG state
- preserve framework and distributed state such as DeepSpeed sharded checkpoints

## Hard Requirements

1. Save full training state, not model-only checkpoints.
- Do not enable options like `save_only_model=true` for long-running training unless the user explicitly accepts losing exact resume.
- Save model, optimizer, scheduler, scaler if used, RNG state, and distributed/framework state needed by the runtime.

2. Resume from the latest complete checkpoint automatically.
- On startup, detect the newest complete checkpoint under the resume directory.
- Resume only from checkpoints that contain every file required by the framework.
- If a checkpoint is incomplete, skip it and log that decision.

3. Handle interruption safely.
- Trap `SIGTERM` and `SIGINT`.
- Forward termination to the trainer so it can finish or flush the current save path.
- Emit the latest resumable checkpoint path during shutdown.

4. Keep checkpoints on durable storage.
- Checkpoint trees and any checkpoint-writing `output_dir` must be on durable storage such as `/scratch`, not ephemeral local scratch and not mixed into lightweight repo logs.

## Implementation Checklist

- Launcher defaults:
  - full-state checkpoint saving enabled
  - periodic save interval defined
  - resume path configurable but defaulting to the task checkpoint directory
- Entry script:
  - accepts `resume_from_checkpoint`
  - passes it into the trainer/runtime
  - does not silently ignore resume
- Checkpoint validation:
  - verify the framework-specific state files exist before resuming
  - examples: optimizer, scheduler, RNG, trainer state, DeepSpeed shard state
- Logging:
  - print startup resume source
  - print skipped incomplete checkpoints
  - print latest resumable checkpoint on shutdown

## Framework Guidance

- Hugging Face Trainer / TRL:
  - prefer `trainer.train(resume_from_checkpoint=...)`
  - keep full checkpoint directories; do not rely on `save_model()` alone
- DeepSpeed:
  - require optimizer/model shard state under the checkpoint step directory
  - do not call a checkpoint "resumable" if only safetensors weights exist
- Accelerate or custom loops:
  - persist optimizer, scheduler, scaler, RNG, and dataloader/step counters when applicable

## Review Rule

A training script is not done until exact resume has been checked explicitly. If the script only reloads model weights, describe it as `restart from weights`, not `resume training`.