ModelOpt Quantization-Aware Distillation
QAD is expensive. Run it only when the user explicitly authorizes QAD for the
target model or run, with a loop or GPU-hour budget; ask for the budget when it
is missing. A Day-0, PTQ, evaluation, comparison, or recipe-search request
alone is not authorization to start QAD.
The authorization covers new recipes from a recipe search only when the user
says so.
Confirm the gap
For both frameworks, reuse only validated, comparable BF16/PTQ results and
the exact benchmark configuration from preceding evaluation or recipe
search; run missing, invalid, or non-comparable baselines. Confirm the target
benchmarks and their context-length needs. Stop if the PTQ gap to BF16 is
already below 1%.
Follow the supported workflow
Read docs/source/guides/quantization_aware_training_and_distillation.rst
for QAD background and framework selection.
Recommend Megatron Bridge for scalable training and its model-specific tutorials.
Choose the applicable tutorial from examples/megatron_bridge/tutorials/README.md.
Honor an explicit HF choice.
Use the common skill's environment-setup.md and workspace-management.md;
for Slurm, also read slurm-setup.md and, when remote, remote-execution.md.
Treat the example README and --help output as authoritative for mutable flags,
commands, containers, and checkpoint formats. Both frameworks can run on local
GPUs or Slurm; use the user's requested environment.
Megatron Bridge execution and training policy
For Megatron Bridge, before constructing commands, read:
examples/megatron_bridge/README.md, especially PTQ, data preparation, QAD,
export, and Slurm usage
examples/megatron_bridge/{quantize.py,distill.py} via --help
Execute in this order
Reproduce PTQ and verify compatibility. In the target runtime, require
AutoBridge.can_handle() for the target model and PTQ through quantize.py
to succeed while preserving the exact preceding PTQ config or recipe:
format, layer selection, calibration data/count, sequence length, and seed.
A changed quantization setting is a new PTQ candidate and must be evaluated
before QAD. In the master-rank .quant_summary.txt, require finite positive
amax for enabled static quantizers; accept dynamic/format-defined None
only when the recipe intends it. Treat the summary as rank-local under model
parallelism.
Choose topology explicitly. Derive the smallest fitting node count and
TP/PP/CP/EP from student and teacher architecture, the chosen sequence length,
and available GPU memory. Prefer CP before TP for small long-context models;
keep EP=1 for dense models and ETP=1 because the current distill.py
workflow does not support expert tensor parallelism. For MoE require:
DP = world_size / (TP * PP * CP)
EDP = world_size / (EP * PP)
- integral DP/EDP,
num_experts % EP == 0, and
GBS % (MBS * DP) == 0
Static-block NVFP4 recipes require TP=1; use PP for memory. With PP, balance
stages with --num_layers_in_{first,last}_pipeline_stage (the first and last
stages also hold the embedding and output layer) and recompute only where
memory needs it (--recompute_num_layers_per_stage). In the first job's
allocation, time a few iterations of each candidate layout with short
launches before the run starts, and keep the fastest. Give each timing
launch a scratch --output_dir: distill.py resumes from
<output_dir>/checkpoints, so a timing checkpoint there would become the
run's starting point.
Prepare the full capped dataset once. Use suitable user-provided data, or
copy examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml as the
default. Set the target tokenizer and workspace path, then materialize the
randomly sampled subset before training. Pack the chosen sequence length;
Megatron's 99,1,0 split creates the 1% validation holdout from the same
data. Keep every source mixed across training, never ordered as a
curriculum, so early stops see the whole blend.
Run and monitor QAD. Run one training job at a time per QAD run;
separate runs may train in parallel. Fold startup validation into the first
job; do not submit separate GPU preflight jobs or split at recovery
iterations. Let training continue while evaluating saved
checkpoints, and cancel it when a stop condition below is met.
Default training policy
| Setting |
Default |
| Sequence length |
32768; adjust for target benchmarks |
| Peak / minimum LR |
1e-5 / 1e-6 |
| LR schedule |
cosine |
LR horizon (train_iters) |
1000 iterations |
| Global batch size |
512 |
| Dataset |
nvidia/Nemotron-Cascade-2-SFT-Data by default |
| Materialized token budget |
17.3B at 32K; cover train_iters at the chosen length |
| Training validation |
every 25 iterations; deterministic 1% holdout; 2 batches |
| Checkpoint interval |
50 iterations |
| Loss logging |
every 10 iterations |
| Recovery benchmark |
from the KD-loss knee, then every checkpoint |
| Slurm duration exit |
220 minutes for a 4-hour allocation |
Run policy
train_iters sets the cosine LR horizon and is a loop knob: with 1000, the
LR stays near its peak through iteration 150. distill.py also stops
training there, with or without exit_interval. Stop runs by the decisions
below and leave exit_interval unset. A run that reaches train_iters while
the decision is still to continue ends as a Miss from under-recovery.
- On Slurm, use initial step timing to submit only enough sequential jobs to reach
a few checkpoints past the expected KD-loss knee; never submit through
train_iters upfront. At each recovery checkpoint, submit to the next only
after its targeted evaluation and any triggered full suite, and only while
the decision is to continue.
- On Slurm, give all training jobs the same run-specific job name and
--dependency=singleton; record job IDs and, on any stop, cancel pending jobs
before the active job.
- Cancel on non-finite loss, repeated skipped iterations, or a sustained spike.
At iteration 50, require the loss aggregate to be lower than at iteration 10.
- At each recovery checkpoint, first evaluate the one to three benchmarks with
the largest PTQ drops or that failed in earlier loops. Run the remaining
original PTQ suite at that checkpoint once the targeted benchmarks are within
run noise of 1%, or after recovery beyond run noise.
- Start benchmark evaluation at the knee of the KD loss curve, where its steep
drop ends, rounded to the next checkpoint, then evaluate every checkpoint
while the scores still change. Before the knee, use loss only to catch
drift; after it, judge and pick checkpoints by benchmarks alone, since KD
loss keeps falling after they peak. Between near ties, the benchmark most
sensitive to quantization decides.
- Run noise is 2x the standard error of a benchmark's gap to BF16: paired per
question when both sides score the same questions, otherwise from both
sides' per-sample standard errors, and pooled across runs once repeats
exist. Repeat a benchmark only when its gap is within run noise of 1%, up to
8 runs per side; the compare-results run floors still apply before a Pass.
- Export each checkpoint to evaluate and pass the ptq skill's
references/checkpoint-validation.md gate on it before evaluation. Run the
serving canary on the run's first export only; later exports change only
weights, so on those the gate passes without the canary: record its
serving-canary row as skipped; first export <path> passed.
- At each evaluated checkpoint, decide:
- Pass: every benchmark of the full original PTQ suite is within 1% of
BF16, each at its compare-results run floor (such as SciCode's 8 runs per
side). Keep the checkpoint and stop the run.
- Miss: two evaluated checkpoints past the knee bring no gain beyond run
noise, or two consecutive checkpoints decline beyond it. Stop and change
one QAD knob in the recipe loop.
- Otherwise continue to the next checkpoint.
- After a duration exit, resume the latest QAD checkpoint in the same output
directory with unchanged prepared data paths/cache, seed, topology, optimizer,
scheduler, iteration, and consumed-sample state; do not restart from PTQ.
- Report, per loop, the PTQ recipe and calibration, data sample, QAD settings,
training topology, loss/state, checkpoints, and comparable BF16/PTQ/QAD
results.
Recipe loop
Each run trains one PTQ recipe and differs from that recipe's control in one
QAD knob. Start from the best PTQ recipe of the preceding PTQ evaluation or
recipe search; its control uses the default policy. When GPUs allow, start
one-knob variants as separate runs alongside the control instead of waiting
for its miss. This loop chooses the QAD settings; the quant-recipe-search
skill chooses the next recipe from its outcome. Stop when the QAD budget is
spent and report the best result so far.
- On a miss, change one QAD knob. Read each missed benchmark against PTQ:
worse than PTQ is drift (lower the peak LR or add matching data); better
than PTQ but short is under-recovery (keep the LR high longer with a larger
train_iters). Knobs, in order of priority when GPUs are limited: peak LR,
train_iters, data blend, then teacher-generated data. A changed train_iters is a fresh run from PTQ, with
data materialized to cover it: resuming would change the decay length the
checkpoint's LR scheduler expects.
- Compare a knob with its control at the same checkpoints past the knee,
on benchmark scores pooled over fresh runs. A rejection holds only for that
PTQ recipe: rerun the control when the recipe changes.
- End the loop on a recipe with a pass, or with a clear miss once every
knob has missed or as soon as QAD stays flat against PTQ or far from
passing. Return the outcome, the best checkpoint, and its scores.
Hugging Face execution and training policy
Before constructing commands, read examples/llm_qat/README.md, its
ARGUMENTS.md, selected training/dataset/Accelerate configs, and
{quantize.py,train.py,export.py} --help.
Follow its quantize → distill → export flow: the PTQ checkpoint is the student;
the original unquantized model is the teacher. Adapt the QAD training and
Accelerate configs to the model and budget.
Resolve tutorial-relative commands and config paths from examples/llm_qat/.
Use the HF tutorial's evaluation and checkpoint resume guidance.
Reuse the Bridge procedure's PTQ-recipe preservation and one-training-job-at-a-time
guidance. Apply the Run policy's loss-stability and
benchmark-recovery stop conditions and report the same fields.
Use HF evaluation/checkpoint intervals; Bridge-specific topology, data defaults,
iteration thresholds, and Slurm job chaining are not HF requirements.
1---2name: modelopt-qad3description: >-4 Run explicitly requested ModelOpt Quantization-Aware Distillation (QAD) through5 Megatron Bridge (recommended) or Hugging Face Transformers to6 recover a measured BF16-to-PTQ accuracy gap.7 Use only when the user explicitly asks for QAD, including its topology, data8 preparation, launch, resume, checkpoint export, or recovery decisions.9metadata:10 source: https://skillmd.com/skills/nvidia/modelopt-qad11---1213# ModelOpt Quantization-Aware Distillation1415QAD is expensive. Run it only when the user explicitly authorizes QAD for the16target model or run, with a loop or GPU-hour budget; ask for the budget when it17is missing. A Day-0, PTQ, evaluation, comparison, or recipe-search request18alone is not authorization to start QAD.19The authorization covers new recipes from a recipe search only when the user20says so.2122## Confirm the gap2324For both frameworks, reuse only validated, comparable BF16/PTQ results and25the exact benchmark configuration from preceding evaluation or recipe26search; run missing, invalid, or non-comparable baselines. Confirm the target27benchmarks and their context-length needs. Stop if the PTQ gap to BF16 is28already below 1%.2930## Follow the supported workflow3132Read `docs/source/guides/quantization_aware_training_and_distillation.rst`33for QAD background and framework selection.34Recommend Megatron Bridge for scalable training and its model-specific tutorials.35Choose the applicable tutorial from `examples/megatron_bridge/tutorials/README.md`.36Honor an explicit HF choice.3738Use the common skill's `environment-setup.md` and `workspace-management.md`;39for Slurm, also read `slurm-setup.md` and, when remote, `remote-execution.md`.4041Treat the example README and `--help` output as authoritative for mutable flags,42commands, containers, and checkpoint formats. Both frameworks can run on local43GPUs or Slurm; use the user's requested environment.4445## Megatron Bridge execution and training policy4647For Megatron Bridge, before constructing commands, read:4849- `examples/megatron_bridge/README.md`, especially PTQ, data preparation, QAD,50 export, and Slurm usage51- `examples/megatron_bridge/{quantize.py,distill.py}` via `--help`5253### Execute in this order54551. **Reproduce PTQ and verify compatibility.** In the target runtime, require56 `AutoBridge.can_handle()` for the target model and PTQ through `quantize.py`57 to succeed while preserving the exact preceding PTQ config or recipe:58 format, layer selection, calibration data/count, sequence length, and seed.59 A changed quantization setting is a new PTQ candidate and must be evaluated60 before QAD. In the master-rank `.quant_summary.txt`, require finite positive61 `amax` for enabled static quantizers; accept `dynamic`/format-defined `None`62 only when the recipe intends it. Treat the summary as rank-local under model63 parallelism.642. **Choose topology explicitly.** Derive the smallest fitting node count and65 TP/PP/CP/EP from student and teacher architecture, the chosen sequence length,66 and available GPU memory. Prefer CP before TP for small long-context models;67 keep EP=1 for dense models and ETP=1 because the current `distill.py`68 workflow does not support expert tensor parallelism. For MoE require:6970 - `DP = world_size / (TP * PP * CP)`71 - `EDP = world_size / (EP * PP)`72 - integral DP/EDP, `num_experts % EP == 0`, and73 `GBS % (MBS * DP) == 0`7475 Static-block NVFP4 recipes require TP=1; use PP for memory. With PP, balance76 stages with `--num_layers_in_{first,last}_pipeline_stage` (the first and last77 stages also hold the embedding and output layer) and recompute only where78 memory needs it (`--recompute_num_layers_per_stage`). In the first job's79 allocation, time a few iterations of each candidate layout with short80 launches before the run starts, and keep the fastest. Give each timing81 launch a scratch `--output_dir`: `distill.py` resumes from82 `<output_dir>/checkpoints`, so a timing checkpoint there would become the83 run's starting point.84853. **Prepare the full capped dataset once.** Use suitable user-provided data, or86 copy `examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml` as the87 default. Set the target tokenizer and workspace path, then materialize the88 randomly sampled subset before training. Pack the chosen sequence length;89 Megatron's `99,1,0` split creates the 1% validation holdout from the same90 data. Keep every source mixed across training, never ordered as a91 curriculum, so early stops see the whole blend.924. **Run and monitor QAD.** Run one training job at a time per QAD run;93 separate runs may train in parallel. Fold startup validation into the first94 job; do not submit separate GPU preflight jobs or split at recovery95 iterations. Let training continue while evaluating saved96 checkpoints, and cancel it when a stop condition below is met.9798### Default training policy99100| Setting | Default |101| --- | --- |102| Sequence length | 32768; adjust for target benchmarks |103| Peak / minimum LR | `1e-5` / `1e-6` |104| LR schedule | cosine |105| LR horizon (`train_iters`) | 1000 iterations |106| Global batch size | 512 |107| Dataset | `nvidia/Nemotron-Cascade-2-SFT-Data` by default |108| Materialized token budget | 17.3B at 32K; cover `train_iters` at the chosen length |109| Training validation | every 25 iterations; deterministic 1% holdout; 2 batches |110| Checkpoint interval | 50 iterations |111| Loss logging | every 10 iterations |112| Recovery benchmark | from the KD-loss knee, then every checkpoint |113| Slurm duration exit | 220 minutes for a 4-hour allocation |114115### Run policy116117- `train_iters` sets the cosine LR horizon and is a loop knob: with 1000, the118 LR stays near its peak through iteration 150. `distill.py` also stops119 training there, with or without `exit_interval`. Stop runs by the decisions120 below and leave `exit_interval` unset. A run that reaches `train_iters` while121 the decision is still to continue ends as a Miss from under-recovery.122- On Slurm, use initial step timing to submit only enough sequential jobs to reach123 a few checkpoints past the expected KD-loss knee; never submit through124 `train_iters` upfront. At each recovery checkpoint, submit to the next only125 after its targeted evaluation and any triggered full suite, and only while126 the decision is to continue.127- On Slurm, give all training jobs the same run-specific job name and128 `--dependency=singleton`; record job IDs and, on any stop, cancel pending jobs129 before the active job.130- Cancel on non-finite loss, repeated skipped iterations, or a sustained spike.131 At iteration 50, require the loss aggregate to be lower than at iteration 10.132- At each recovery checkpoint, first evaluate the one to three benchmarks with133 the largest PTQ drops or that failed in earlier loops. Run the remaining134 original PTQ suite at that checkpoint once the targeted benchmarks are within135 run noise of 1%, or after recovery beyond run noise.136- Start benchmark evaluation at the knee of the KD loss curve, where its steep137 drop ends, rounded to the next checkpoint, then evaluate every checkpoint138 while the scores still change. Before the knee, use loss only to catch139 drift; after it, judge and pick checkpoints by benchmarks alone, since KD140 loss keeps falling after they peak. Between near ties, the benchmark most141 sensitive to quantization decides.142- Run noise is 2x the standard error of a benchmark's gap to BF16: paired per143 question when both sides score the same questions, otherwise from both144 sides' per-sample standard errors, and pooled across runs once repeats145 exist. Repeat a benchmark only when its gap is within run noise of 1%, up to146 8 runs per side; the compare-results run floors still apply before a Pass.147- Export each checkpoint to evaluate and pass the ptq skill's148 `references/checkpoint-validation.md` gate on it before evaluation. Run the149 serving canary on the run's first export only; later exports change only150 weights, so on those the gate passes without the canary: record its151 serving-canary row as `skipped; first export <path> passed`.152- At each evaluated checkpoint, decide:153 - **Pass:** every benchmark of the full original PTQ suite is within 1% of154 BF16, each at its compare-results run floor (such as SciCode's 8 runs per155 side). Keep the checkpoint and stop the run.156 - **Miss:** two evaluated checkpoints past the knee bring no gain beyond run157 noise, or two consecutive checkpoints decline beyond it. Stop and change158 one QAD knob in the [recipe loop](#recipe-loop).159 - **Otherwise continue** to the next checkpoint.160- After a duration exit, resume the latest QAD checkpoint in the same output161 directory with unchanged prepared data paths/cache, seed, topology, optimizer,162 scheduler, iteration, and consumed-sample state; do not restart from PTQ.163- Report, per loop, the PTQ recipe and calibration, data sample, QAD settings,164 training topology, loss/state, checkpoints, and comparable BF16/PTQ/QAD165 results.166167### Recipe loop168169Each run trains one PTQ recipe and differs from that recipe's control in one170QAD knob. Start from the best PTQ recipe of the preceding PTQ evaluation or171recipe search; its control uses the default policy. When GPUs allow, start172one-knob variants as separate runs alongside the control instead of waiting173for its miss. This loop chooses the QAD settings; the quant-recipe-search174skill chooses the next recipe from its outcome. Stop when the QAD budget is175spent and report the best result so far.1761771. **On a miss, change one QAD knob.** Read each missed benchmark against PTQ:178 worse than PTQ is drift (lower the peak LR or add matching data); better179 than PTQ but short is under-recovery (keep the LR high longer with a larger180 `train_iters`). Knobs, in order of priority when GPUs are limited: peak LR,181 `train_iters`, data blend, then teacher-generated data. A changed `train_iters` is a fresh run from PTQ, with182 data materialized to cover it: resuming would change the decay length the183 checkpoint's LR scheduler expects.1842. **Compare a knob with its control at the same checkpoints** past the knee,185 on benchmark scores pooled over fresh runs. A rejection holds only for that186 PTQ recipe: rerun the control when the recipe changes.1873. **End the loop on a recipe** with a pass, or with a clear miss once every188 knob has missed or as soon as QAD stays flat against PTQ or far from189 passing. Return the outcome, the best checkpoint, and its scores.190191## Hugging Face execution and training policy192193Before constructing commands, read `examples/llm_qat/README.md`, its194`ARGUMENTS.md`, selected training/dataset/Accelerate configs, and195`{quantize.py,train.py,export.py} --help`.196Follow its quantize → distill → export flow: the PTQ checkpoint is the student;197the original unquantized model is the teacher. Adapt the QAD training and198Accelerate configs to the model and budget.199Resolve tutorial-relative commands and config paths from `examples/llm_qat/`.200Use the HF tutorial's evaluation and checkpoint resume guidance.201202Reuse the Bridge procedure's PTQ-recipe preservation and one-training-job-at-a-time203guidance. Apply the [Run policy](#run-policy)'s loss-stability and204benchmark-recovery stop conditions and report the same fields.205Use HF evaluation/checkpoint intervals; Bridge-specific topology, data defaults,206iteration thresholds, and Slurm job chaining are not HF requirements.