Quant Recipe Search
Use this skill when quantization is an iterative recipe search, not a one-off
PTQ run. The skill owns strategy: define success, choose the search space,
sequence candidates, and decide the next iteration. It delegates checkpoint
generation, serving, evaluation, monitoring, and metric comparison to the
existing execution skills.
Treat a direct request such as "find the best quantization recipe and generate a
PTQ checkpoint for this model" as enough to start. Recover local state first,
then ask only for missing decisions that change the search.
Skill Boundaries
- Use
modelopt-ptq to produce and validate checkpoints.
- Use
modelopt-deployment to serve checkpoints and debug serving-specific flags.
- Use
modelopt-evaluation to create NEL configs and submit evals.
- Use
modelopt-launching-evals to run, resume, debug, and analyze NEL runs.
- Use
modelopt-monitor for active job tracking.
- Use
modelopt-accessing-mlflow for MLflow artifact lookup.
- Use
modelopt-compare-results for validated baseline-vs-candidate deltas and score-field comparability.
Do not duplicate those workflows here. This skill should leave the user with a clear recipe portfolio, success metric, experiment sequence, and next decision.
Problem
The task is to find the best recipe for a user-defined target, not merely to
produce a quantized checkpoint. A generated PTQ checkpoint is only a candidate.
It becomes a recommended recipe only after evaluation and comparison against the
matching baseline.
Required inputs before planning candidates:
- Optimization goal: compute/throughput, memory/latency, or a custom metric.
- Primary quantization family: for example NVFP4, W4A16 NVFP4, FP8/W8A8,
INT4/AWQ, or a custom mixed set.
- Benchmark set or baseline results: the user-defined acceptance surface.
If any of these are missing, ask for them. Do not silently default to FP8/W8A8
or call a checkpoint "best" before evaluation.
Default success rule: maximize the chosen performance objective while keeping
each benchmark within 1 percentage point of the matching BF16/FP16 baseline.
Near-threshold or noisy regressions require reruns before making a decision.
Search Space
Keep the search space explicit. A candidate recipe is a tuple across these axes:
- Numeric format: FP8/W8A8, NVFP4/W4A4, W4A16 NVFP4, INT4/AWQ, or mixed
formats such as NVFP4+FP8.
- Calibration/search algorithm: max calibration, MSE calibration, GPTQ,
AWQ, AutoQuant scoring, and calibration dataset or sample-count variants.
- Selection method: manual/heuristic rules, sensitivity-guided manual
recipes, AutoQuant selection, or a hybrid of AutoQuant plus manual overrides.
- Module family: attention, MLP, MoE experts, routers/gates, embeddings,
lm_head, adapters, vision encoders, and model-specific modules.
- Layer position: first/last transformer-layer counts or explicit ordinal
ranges to keep in BF16. First 3-4 and last 1-2 layers are common starting
candidate ranges, not defaults.
- Runtime fusion constraints: modules fused by the inference library must
use compatible quantization. Examples: vLLM Qwen
linear_attn.in_proj_qkvz
and fused MoE expert projections such as gate/up (w1/w3).
- Calibration budget: dataset mix, sample count, sequence length, and batch
settings.
Do not collapse the search to one dimension such as numeric format only. Read
references/recipe_iteration.md when choosing concrete axes or candidates.
Design Workflow
Recover state
- Read result tables, recipe logs, AutoQuant states, sensitivity reports, and
experiment notes before proposing new work.
- Ask
modelopt-monitor, modelopt-launching-evals, or modelopt-compare-results to recover active
job state and completed metrics when needed.
Define the target
- Confirm the optimization goal, primary quantization family, benchmark set,
accuracy-loss threshold, calibration budget, and cost metric.
- Include quantization metadata such as scale storage in active-cost or size
estimates.
Pick baselines and first candidates
- Always include BF16/FP16 and a near-lossless FP8/W8A8 baseline unless FP8
itself is the target.
- For ModelOpt work, start from
modelopt_recipes: model-specific recipes
first, then general PTQ presets or recipe fragments.
- Add an AutoQuant candidate in the requested primary family when AutoQuant
is available. Expect AutoQuant to find a better trade-off than a first
manual recipe, but validate that assumption with the same evals.
- Add at least one manual or sensitivity-guided candidate so AutoQuant can be
compared against controlled ablations and there is a fallback if AutoQuant
misses the best frontier or hits runtime constraints.
- When sensitivity or model behavior implicates boundary layers, add a
controlled first-layer, last-layer, or combined BF16 exclusion candidate.
Do not preserve boundary layers without testing the trade-off.
Generate candidates
- Delegate checkpoint generation and PTQ validation to
modelopt-ptq.
- Change one major axis at a time: format, calibration algorithm, module
family, layer position, granularity, or calibration data.
- Use AutoQuant for broad candidate generation and sensitivity reports; use
manual recipes for controlled module-family ablations and overrides.
- Resolve positional ordinals against the model's transformer block sequence.
For manual recipes, add those blocks to the recipe exclusions. For
AutoQuant or hybrid candidates, pass positional exclusions into the
selected implementation when supported; otherwise apply a manual override
to its result and record the limitation.
Gate before scaling
- Validate checkpoint coverage and metadata.
- Reject or rewrite recipes that mix quantization algorithms inside a fused
runtime group.
- Ensure positional exclusions preserve complete fused runtime groups, then
evaluate the candidate against the same BF16 baseline and acceptance
criteria as every other recipe.
- If the checkpoint is valid but serving fails due to runtime support, do not
reject the recipe immediately. Delegate to
modelopt-deployment / modelopt-debug for small
patches or flags, then rerun a pipe-clean check.
Iteration Loop
- Run cheap screen evals for every candidate that passes the gates.
- Compare accuracy, verbosity/token usage, and active cost against baselines.
- Rerun noisy or near-threshold results before labeling a regression.
- Decide the next candidate:
- Accuracy drop: protect or ablate sensitive module families, try MSE/GPTQ,
use AutoQuant sensitivity to choose overrides, or test first/last-layer
BF16 exclusions when evidence points to boundary sensitivity.
- Poor performance/cost: quantize the next high-cost active family, adjust
active-cost objective, or try a more aggressive format.
- AutoQuant underperforms manual recipes: inspect sensitivity reports,
achieved bits, excluded modules, and runtime-fusion constraints; keep the
manual recipe in the portfolio instead of forcing the AutoQuant result.
- Runtime incompatibility: rewrite around fused groups or isolate deployment
support from checkpoint quality.
- Repeated AutoQuant recipes: inspect achieved bits and recipe hashes, then
adjust constraints before launching a larger sweep.
- Promote only when
modelopt-compare-results shows no failed external sanity check,
the candidate is comparable to the validated measured baseline, and the
user-defined goal is met. An externally unverified baseline is non-blocking.
When the user has authorized QAD, a candidate is a recipe plus the QAD settings
from the modelopt-qad skill's recipe loop, which ends each recipe with a pass or a
clear miss. A PTQ recipe that already meets the threshold counts as a pass.
Step the layout from that outcome:
- After a pass, add one module group to the passing recipe, in order of
savings on the success metric; the MTP head is its own group. A group can be
a slice: its middle layers with the first and last few kept, or some of its
projections (such as GDN
in_proj only, or attention linear_qkv only).
Find and evaluate the new layout's best PTQ recipe, starting from the
passing recipe's calibration, before its QAD. Step one layout at a time;
the modelopt-qad recipe loop runs its knob variants in parallel.
- After a clear miss, shrink the added group to a slice; if that misses too,
drop the group and try the next one on the last passing recipe. If no recipe
has passed yet, report no QAD pass and return to the accuracy-drop steps
above.
- Stop when every group has been added or dropped, or the QAD budget is spent.
Return the passing recipe that best meets the success metric, with its QAD
checkpoint.
Maintain a recipe portfolio table with recipe name, objective, active-cost
estimate, calibration notes, checkpoint path, eval/log references, accuracy,
verbosity, positional exclusions, and decision.
References
- For recipe design, search-space details, sensitivity, and active-cost
accounting, read
references/recipe_iteration.md.
- For a concrete prior case study, read
references/qwen36_case_study.md only
when Qwen3.5/Qwen3.6 details are relevant.
1---2name: modelopt-quant-recipe-search3description: >-4 Use when the user asks to find, search for, or optimize the best quantization5 recipe for a model, including direct requests like "find the best quantization6 recipe and generate a PTQ checkpoint." Guides the multi-candidate loop:7 choose compute-vs-memory success metrics, select ModelOpt recipe baselines,8 design AutoQuant/manual recipe deltas, interpret sensitivity, and decide next9 candidates. Do NOT use for a single known PTQ recipe run (use modelopt-ptq), serving10 (use modelopt-deployment), creating/running evals (use modelopt-evaluation or modelopt-launching-evals),11 monitoring jobs (use modelopt-monitor), MLflow browsing (use modelopt-accessing-mlflow), or12 comparing completed baseline-vs-candidate scores only (use modelopt-compare-results).13metadata:14 source: https://skillmd.com/skills/nvidia/modelopt-quant-recipe-search15---1617# Quant Recipe Search1819Use this skill when quantization is an iterative recipe search, not a one-off20PTQ run. The skill owns strategy: define success, choose the search space,21sequence candidates, and decide the next iteration. It delegates checkpoint22generation, serving, evaluation, monitoring, and metric comparison to the23existing execution skills.2425Treat a direct request such as "find the best quantization recipe and generate a26PTQ checkpoint for this model" as enough to start. Recover local state first,27then ask only for missing decisions that change the search.2829## Skill Boundaries3031- Use `modelopt-ptq` to produce and validate checkpoints.32- Use `modelopt-deployment` to serve checkpoints and debug serving-specific flags.33- Use `modelopt-evaluation` to create NEL configs and submit evals.34- Use `modelopt-launching-evals` to run, resume, debug, and analyze NEL runs.35- Use `modelopt-monitor` for active job tracking.36- Use `modelopt-accessing-mlflow` for MLflow artifact lookup.37- Use `modelopt-compare-results` for validated baseline-vs-candidate deltas and score-field comparability.3839Do not duplicate those workflows here. This skill should leave the user with a clear recipe portfolio, success metric, experiment sequence, and next decision.4041## Problem4243The task is to find the best recipe for a user-defined target, not merely to44produce a quantized checkpoint. A generated PTQ checkpoint is only a candidate.45It becomes a recommended recipe only after evaluation and comparison against the46matching baseline.4748Required inputs before planning candidates:4950- **Optimization goal:** compute/throughput, memory/latency, or a custom metric.51- **Primary quantization family:** for example NVFP4, W4A16 NVFP4, FP8/W8A8,52 INT4/AWQ, or a custom mixed set.53- **Benchmark set or baseline results:** the user-defined acceptance surface.5455If any of these are missing, ask for them. Do not silently default to FP8/W8A856or call a checkpoint "best" before evaluation.5758Default success rule: maximize the chosen performance objective while keeping59each benchmark within 1 percentage point of the matching BF16/FP16 baseline.60Near-threshold or noisy regressions require reruns before making a decision.6162## Search Space6364Keep the search space explicit. A candidate recipe is a tuple across these axes:6566- **Numeric format:** FP8/W8A8, NVFP4/W4A4, W4A16 NVFP4, INT4/AWQ, or mixed67 formats such as NVFP4+FP8.68- **Calibration/search algorithm:** max calibration, MSE calibration, GPTQ,69 AWQ, AutoQuant scoring, and calibration dataset or sample-count variants.70- **Selection method:** manual/heuristic rules, sensitivity-guided manual71 recipes, AutoQuant selection, or a hybrid of AutoQuant plus manual overrides.72- **Module family:** attention, MLP, MoE experts, routers/gates, embeddings,73 `lm_head`, adapters, vision encoders, and model-specific modules.74- **Layer position:** first/last transformer-layer counts or explicit ordinal75 ranges to keep in BF16. First 3-4 and last 1-2 layers are common starting76 candidate ranges, not defaults.77- **Runtime fusion constraints:** modules fused by the inference library must78 use compatible quantization. Examples: vLLM Qwen `linear_attn.in_proj_qkvz`79 and fused MoE expert projections such as gate/up (`w1`/`w3`).80- **Calibration budget:** dataset mix, sample count, sequence length, and batch81 settings.8283Do not collapse the search to one dimension such as numeric format only. Read84`references/recipe_iteration.md` when choosing concrete axes or candidates.8586## Design Workflow87881. **Recover state**89 - Read result tables, recipe logs, AutoQuant states, sensitivity reports, and90 experiment notes before proposing new work.91 - Ask `modelopt-monitor`, `modelopt-launching-evals`, or `modelopt-compare-results` to recover active92 job state and completed metrics when needed.93942. **Define the target**95 - Confirm the optimization goal, primary quantization family, benchmark set,96 accuracy-loss threshold, calibration budget, and cost metric.97 - Include quantization metadata such as scale storage in active-cost or size98 estimates.991003. **Pick baselines and first candidates**101 - Always include BF16/FP16 and a near-lossless FP8/W8A8 baseline unless FP8102 itself is the target.103 - For ModelOpt work, start from `modelopt_recipes`: model-specific recipes104 first, then general PTQ presets or recipe fragments.105 - Add an AutoQuant candidate in the requested primary family when AutoQuant106 is available. Expect AutoQuant to find a better trade-off than a first107 manual recipe, but validate that assumption with the same evals.108 - Add at least one manual or sensitivity-guided candidate so AutoQuant can be109 compared against controlled ablations and there is a fallback if AutoQuant110 misses the best frontier or hits runtime constraints.111 - When sensitivity or model behavior implicates boundary layers, add a112 controlled first-layer, last-layer, or combined BF16 exclusion candidate.113 Do not preserve boundary layers without testing the trade-off.1141154. **Generate candidates**116 - Delegate checkpoint generation and PTQ validation to `modelopt-ptq`.117 - Change one major axis at a time: format, calibration algorithm, module118 family, layer position, granularity, or calibration data.119 - Use AutoQuant for broad candidate generation and sensitivity reports; use120 manual recipes for controlled module-family ablations and overrides.121 - Resolve positional ordinals against the model's transformer block sequence.122 For manual recipes, add those blocks to the recipe exclusions. For123 AutoQuant or hybrid candidates, pass positional exclusions into the124 selected implementation when supported; otherwise apply a manual override125 to its result and record the limitation.1261275. **Gate before scaling**128 - Validate checkpoint coverage and metadata.129 - Reject or rewrite recipes that mix quantization algorithms inside a fused130 runtime group.131 - Ensure positional exclusions preserve complete fused runtime groups, then132 evaluate the candidate against the same BF16 baseline and acceptance133 criteria as every other recipe.134 - If the checkpoint is valid but serving fails due to runtime support, do not135 reject the recipe immediately. Delegate to `modelopt-deployment` / `modelopt-debug` for small136 patches or flags, then rerun a pipe-clean check.137138## Iteration Loop1391401. Run cheap screen evals for every candidate that passes the gates.1412. Compare accuracy, verbosity/token usage, and active cost against baselines.1423. Rerun noisy or near-threshold results before labeling a regression.1434. Decide the next candidate:144 - Accuracy drop: protect or ablate sensitive module families, try MSE/GPTQ,145 use AutoQuant sensitivity to choose overrides, or test first/last-layer146 BF16 exclusions when evidence points to boundary sensitivity.147 - Poor performance/cost: quantize the next high-cost active family, adjust148 active-cost objective, or try a more aggressive format.149 - AutoQuant underperforms manual recipes: inspect sensitivity reports,150 achieved bits, excluded modules, and runtime-fusion constraints; keep the151 manual recipe in the portfolio instead of forcing the AutoQuant result.152 - Runtime incompatibility: rewrite around fused groups or isolate deployment153 support from checkpoint quality.154 - Repeated AutoQuant recipes: inspect achieved bits and recipe hashes, then155 adjust constraints before launching a larger sweep.1565. Promote only when `modelopt-compare-results` shows no failed external sanity check,157 the candidate is comparable to the validated measured baseline, and the158 user-defined goal is met. An externally unverified baseline is non-blocking.159160When the user has authorized QAD, a candidate is a recipe plus the QAD settings161from the `modelopt-qad` skill's recipe loop, which ends each recipe with a pass or a162clear miss. A PTQ recipe that already meets the threshold counts as a pass.163Step the layout from that outcome:164165- After a pass, add one module group to the passing recipe, in order of166 savings on the success metric; the MTP head is its own group. A group can be167 a slice: its middle layers with the first and last few kept, or some of its168 projections (such as GDN `in_proj` only, or attention `linear_qkv` only).169 Find and evaluate the new layout's best PTQ recipe, starting from the170 passing recipe's calibration, before its QAD. Step one layout at a time;171 the `modelopt-qad` recipe loop runs its knob variants in parallel.172- After a clear miss, shrink the added group to a slice; if that misses too,173 drop the group and try the next one on the last passing recipe. If no recipe174 has passed yet, report no QAD pass and return to the accuracy-drop steps175 above.176- Stop when every group has been added or dropped, or the QAD budget is spent.177 Return the passing recipe that best meets the success metric, with its QAD178 checkpoint.179180Maintain a recipe portfolio table with recipe name, objective, active-cost181estimate, calibration notes, checkpoint path, eval/log references, accuracy,182verbosity, positional exclusions, and decision.183184## References185186- For recipe design, search-space details, sensitivity, and active-cost187 accounting, read `references/recipe_iteration.md`.188- For a concrete prior case study, read `references/qwen36_case_study.md` only189 when Qwen3.5/Qwen3.6 details are relevant.