Train or distill with FastGen
Apply fastgen-runtime first. The
README and the configuration README
own launch commands, run directories and checkpoint storage; the
troubleshooting guide owns failure diagnosis. Read
run contracts to bound a run, read its logs or continue a
checkpoint.
- Goal and inputs. Decide whether the deliverable is a smoke test, a bounded
experiment, a quality target or a resumed run. Use
fastgen-configure for the recipe and
fastgen-dataset for data; check every enabled
loss, and the validation loader, against the real-data versus prompt-only contract.
- Weights versus state. Distinguish external weights, warm starts and full resume;
the checkpoint table
lists the settings and their order. Resume is on by default, so independent runs need
new run identities, and FSDP runs resume only from sharded checkpoints
(checkpoint storage).
- Bound and observe. Before launching set
trainer.max_iter,
trainer.save_ckpt_iter, trainer.validation_iter, trainer.logging_iter and the
W&B destination; keep the algorithmic callbacks. For a smoke test log scalars and media
every iteration through the W&B callback and run
long enough to cover a student update and every alternating optimizer phase.
- Launch. Enable exactly one of
trainer.ddp/trainer.fsdp for multi-GPU runs; a
launcher alone distributes nothing. Budget memory for teacher, student, auxiliary
networks, optimizer state and preprocessors
(CUDA out of memory),
then confirm the runtime startup checks
and, when requested, online tracker uploads.
- Monitor. Check finite losses and gradients, parameter updates and samples with
fastgen-validate; compare fixed
held-out inputs before and after training with
fastgen-inference. Stop on non-finite updates,
broken conditioning or degrading samples before spending the budget.
Verify the run's saved checkpoint and readable media before reporting success; include
these artifact checks in a review-only run plan. Scheduler requeue alone saves nothing;
rely on periodic checkpoints or a
trainer.auto_resume integration (default: no-op).
Report run URLs, the resolved config, executed iterations and optimizer phases,
checkpoint location, compute used and validation evidence. A submitted job is not a
successful run and a smoke test is not convergence. For stalls distinguish queueing,
downloads, empty loaders and distributed startup; for resume mismatches inspect run
identity and checkpoint contents first.
1---2name: fastgen-train3description: >-4 Run, resume or troubleshoot NVIDIA FastGen training: SFT or distillation with train.py, bounded smoke tests, distributed launches, checkpoints, validation and W&B evidence. Use it whenever training is executed or a training run misbehaves.5license: Apache-2.06metadata:7 source: https://skillmd.com/skills/nvidia/fastgen-train8 author: "Julius Berner <jberner@nvidia.com>"9 domain: ml10---1112# Train or distill with FastGen1314Apply [fastgen-runtime](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-runtime/SKILL.md) first. The15[README](https://github.com/NVlabs/FastGen/blob/main/README.md#quick-start) and the [configuration README](https://github.com/NVlabs/FastGen/blob/main/fastgen/configs/README.md)16own launch commands, run directories and checkpoint storage; the17[troubleshooting guide](https://github.com/NVlabs/FastGen/blob/main/fern/docs/troubleshooting.mdx) owns failure diagnosis. Read18[run contracts](references/run-contract.md) to bound a run, read its logs or continue a19checkpoint.20211. **Goal and inputs.** Decide whether the deliverable is a smoke test, a bounded22 experiment, a quality target or a resumed run. Use23 [fastgen-configure](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-configure/SKILL.md) for the recipe and24 [fastgen-dataset](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-dataset/SKILL.md) for data; check every enabled25 loss, and the validation loader, against the real-data versus prompt-only contract.262. **Weights versus state.** Distinguish external weights, warm starts and full resume;27 the [checkpoint table](https://github.com/NVlabs/FastGen/blob/main/fern/docs/troubleshooting.mdx#a-checkpoint-fails-to-load)28 lists the settings and their order. Resume is on by default, so independent runs need29 new run identities, and FSDP runs resume only from sharded checkpoints30 ([checkpoint storage](https://github.com/NVlabs/FastGen/blob/main/fastgen/configs/README.md#distributed-checkpoint-storage)).313. **Bound and observe.** Before launching set `trainer.max_iter`,32 `trainer.save_ckpt_iter`, `trainer.validation_iter`, `trainer.logging_iter` and the33 W&B destination; keep the algorithmic callbacks. For a smoke test log scalars and media34 every iteration through the [W&B callback](https://github.com/NVlabs/FastGen/blob/main/fastgen/callbacks/README.md) and run35 long enough to cover a student update and every alternating optimizer phase.364. **Launch.** Enable exactly one of `trainer.ddp`/`trainer.fsdp` for multi-GPU runs; a37 launcher alone distributes nothing. Budget memory for teacher, student, auxiliary38 networks, optimizer state and preprocessors39 ([CUDA out of memory](https://github.com/NVlabs/FastGen/blob/main/fern/docs/troubleshooting.mdx#cuda-runs-out-of-memory)),40 then confirm the [runtime startup checks](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-runtime/SKILL.md)41 and, when requested, online tracker uploads.425. **Monitor.** Check finite losses and gradients, parameter updates and samples with43 [fastgen-validate](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-validate/SKILL.md); compare fixed44 held-out inputs before and after training with45 [fastgen-inference](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-inference/SKILL.md). Stop on non-finite updates,46 broken conditioning or degrading samples before spending the budget.47 Verify the run's saved checkpoint and readable media before reporting success; include48 these artifact checks in a review-only run plan. Scheduler requeue alone saves nothing;49 rely on periodic checkpoints or a `trainer.auto_resume` integration (default: no-op).5051Report run URLs, the resolved config, executed iterations and optimizer phases,52checkpoint location, compute used and validation evidence. A submitted job is not a53successful run and a smoke test is not convergence. For stalls distinguish queueing,54downloads, empty loaders and distributed startup; for resume mismatches inspect run55identity and checkpoint contents first.