# Fastgen Train

> Run, resume or troubleshoot NVIDIA FastGen training: SFT or distillation with train.py, bounded smoke tests, distributed launches, checkpoints, validation and W&B evidence. Use it whenever training is executed or a training run misbehaves.

- Skill: `nvidia/fastgen-train` (Agent Skill, multi-file: 13 files)
- Install (CLI): `npx get-skill add nvidia/fastgen-train`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia/fastgen-train/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: Apache-2.0
- Author: NVIDIA (https://skillmd.com/u/nvidia), verified publisher
- Updated: 2026-11-11
- Page: https://skillmd.com/skills/nvidia/fastgen-train

---


# Train or distill with FastGen

Apply [fastgen-runtime](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-runtime/SKILL.md) first. The
[README](https://github.com/NVlabs/FastGen/blob/main/README.md#quick-start) and the [configuration README](https://github.com/NVlabs/FastGen/blob/main/fastgen/configs/README.md)
own launch commands, run directories and checkpoint storage; the
[troubleshooting guide](https://github.com/NVlabs/FastGen/blob/main/fern/docs/troubleshooting.mdx) owns failure diagnosis. Read
[run contracts](references/run-contract.md) to bound a run, read its logs or continue a
checkpoint.

1. **Goal and inputs.** Decide whether the deliverable is a smoke test, a bounded
   experiment, a quality target or a resumed run. Use
   [fastgen-configure](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-configure/SKILL.md) for the recipe and
   [fastgen-dataset](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-dataset/SKILL.md) for data; check every enabled
   loss, and the validation loader, against the real-data versus prompt-only contract.
2. **Weights versus state.** Distinguish external weights, warm starts and full resume;
   the [checkpoint table](https://github.com/NVlabs/FastGen/blob/main/fern/docs/troubleshooting.mdx#a-checkpoint-fails-to-load)
   lists the settings and their order. Resume is on by default, so independent runs need
   new run identities, and FSDP runs resume only from sharded checkpoints
   ([checkpoint storage](https://github.com/NVlabs/FastGen/blob/main/fastgen/configs/README.md#distributed-checkpoint-storage)).
3. **Bound and observe.** Before launching set `trainer.max_iter`,
   `trainer.save_ckpt_iter`, `trainer.validation_iter`, `trainer.logging_iter` and the
   W&B destination; keep the algorithmic callbacks. For a smoke test log scalars and media
   every iteration through the [W&B callback](https://github.com/NVlabs/FastGen/blob/main/fastgen/callbacks/README.md) and run
   long enough to cover a student update and every alternating optimizer phase.
4. **Launch.** Enable exactly one of `trainer.ddp`/`trainer.fsdp` for multi-GPU runs; a
   launcher alone distributes nothing. Budget memory for teacher, student, auxiliary
   networks, optimizer state and preprocessors
   ([CUDA out of memory](https://github.com/NVlabs/FastGen/blob/main/fern/docs/troubleshooting.mdx#cuda-runs-out-of-memory)),
   then confirm the [runtime startup checks](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-runtime/SKILL.md)
   and, when requested, online tracker uploads.
5. **Monitor.** Check finite losses and gradients, parameter updates and samples with
   [fastgen-validate](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-validate/SKILL.md); compare fixed
   held-out inputs before and after training with
   [fastgen-inference](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-inference/SKILL.md). Stop on non-finite updates,
   broken conditioning or degrading samples before spending the budget.
   Verify the run's saved checkpoint and readable media before reporting success; include
   these artifact checks in a review-only run plan. Scheduler requeue alone saves nothing;
   rely on periodic checkpoints or a `trainer.auto_resume` integration (default: no-op).

Report run URLs, the resolved config, executed iterations and optimizer phases,
checkpoint location, compute used and validation evidence. A submitted job is not a
successful run and a smoke test is not convergence. For stalls distinguish queueing,
downloads, empty loaders and distributed startup; for resume mismatches inspect run
identity and checkpoint contents first.

