Vllm Quantization

vLLM quantization — operator skill

Last verified: 2026-08-11. Claims below are probed against v0.27.0 (2026-08-10); latest stable is v0.27.1 (2026-08-11), a one-change patch — see references/sources.md for the per-ref audit table.

For production vLLM operators on H100 / H200 / B200 / B300 / GB200 / GB300 fleets deciding which quantization format fits a given target model, producing a checkpoint vLLM will actually load, wiring the right KV-cache dtype, diagnosing accuracy or throughput regressions after an upgrade, and composing quantization with speculative decoding / LoRA / MoE.

Pointer-map format: this SKILL.md picks the format and CLI; the files in references/ hold the per-format deep dives, exact source pointers, and troubleshooting cards. Follow the link, don't paraphrase from memory — the quantization layer moves faster than any other subsystem in vLLM (six formats landed in v0.19 alone).

When quantization wins, when it doesn't

Quantization trades weight precision for memory + compute:

• KV-capacity bound (long context, high concurrency) — FP8 or NVFP4 KV cache gives a 2×/4× KV-capacity multiplier; weight format matters much less than getting --kv-cache-dtype right. Measure kv_cache_usage_perc.
• Memory-bandwidth bound (small batch, decode-heavy, 70B+ on < 8 GPUs) — weight quantization (NVFP4 / FP8 / W4A16) reduces HBM traffic per token, giving 1.5–3× decode throughput on a well-matched target+kernel.
• Compute bound (prefill, large batch, small model) — quantization may not help; Blackwell FP4 Tensor Cores are the first architecture where W4A4 actually beats FP8 in compute-bound regimes. On Hopper, W4A16 is memory-only — MMA still runs FP16.
• Multi-node EP / disaggregated serving — NVFP4 reduces all-to-all by 4× vs BF16. DeepSeek-R1 / V3.2 on GB200/GB300 gets most of its throughput from NVFP4 over the fabric, not from per-GPU compute (see vLLM WideEP blog).

Quantized models are not equivalent to the BF16 original. Always eval on actual traffic. Stock NVFP4 checkpoints recover ~99 % at 70B+, ~95–98 % at 7B–14B (Red Hat / NVIDIA numbers). Code / math / agentic workloads hit harder.

Format selection — pick once per hardware

Cross-hardware rule of thumb: produce one NVFP4 checkpoint per model. It loads on Blackwell natively and on Hopper via emulation (PR #35733, v0.19). A separate fp8 checkpoint is still worth keeping for older Hopper nodes where the NVFP4 emulation path is slower.

The --quantization flag values (all 31 at v0.27.0)

Ground truth is the QuantizationMethods = Literal[...] block in vllm/model_executor/layers/quantization/__init__.py (get_quantization_config() dispatches below it) — grep the symbol, not a line range. Full catalog with file paths, min-capability, kernel map, and notes: references/formats.md.

Production formats (keep in head):

Deprecated / legacy / narrow: fbgemm_fp8 and fp_quant are the only two in vLLM's own DEPRECATED_QUANTIZATION_METHODS list. Also avoid for new work: awq (unfused Triton — use awq_marlin), gptq (unfused — use gptq_marlin), experts_int8 (use int8_per_channel_weight_only), moe_wna16, bitsandbytes, inc / auto-round (Intel), torchao. Aliases auto_awq / auto_gptq resolve to the same configs as awq / gptq.

Gone from the in-tree flag list — do not offer them:

• gguf — migrated out-of-tree to vllm-gguf-plugin. pip install vllm-gguf-plugin first, then vllm serve <repo>:Q4_K_M --tokenizer <base-model>. Still experimental; not a datacenter path.
• cpu_awq — folded into awq_marlin (PR #43841, merged 2026-05-28). Use awq_marlin on CPU.

Newer values not in older skill copies: humming (Humming mixed-precision, also a --linear-backend / --moe-backend value), deepseek_v4_fp8 (DeepSeek-V4 checkpoint config), fp8_per_channel and nvfp4_per_token (online shorthands; nvfp4_per_token added v0.26.0, #48538).

KV-cache dtypes (all 16 at v0.27.0)

Single dispatch: vllm/config/cache.py — grep the CacheDType = Literal[...] block, not a line range; it has grown every minor.

• auto — match model weight dtype. float16 / bfloat16 — pin an explicit unquantized KV dtype.
• fp8, fp8_e4m3, fp8_e5m2 — the production path. E4M3 is default; E5M2 only for ROCm-specific setups.
• fp8_inc (Intel), fp8_ds_mla (DeepSeek MLA variant).
• int4_per_token_head, int8_per_token_head, fp8_per_token_head — dynamic per-(token,head) scales computed in-kernel. No checkpoint scales needed. Added in PR #34281, v0.17.
• turboquant_k8v4, turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_3bit_nc — Hadamard-rotated 2-4 bit KV (v0.19, PR #38479).
• nvfp4 — shipped, not roadmap. #32220 closed COMPLETED 2026-05-04; v0.25.0 landed NVFP4 KV with skip-layers sliding window (#42890) and it is an accepted CacheDType value at v0.27.0. Backend support is narrow — see references/kv-cache.md.

--calculate-kv-scales was deprecated in v0.19 (PR #37201). Use pre-calibrated scales (LLM Compressor produces them) or let per-token-head scales be computed dynamically.

Producing a checkpoint

vLLM doesn't quantize — a separate tool does, then vLLM loads the result. Two production paths exist:

1. llm-compressor (vLLM-project) — outputs compressed-tensors format. Preferred for the open ecosystem. Covered in references/llm-compressor.md.
2. NVIDIA ModelOpt — outputs ModelOpt HF format, also consumable by TRT-LLM and SGLang. Preferred for NVFP4 on Blackwell. Covered in references/modelopt.md.

Quick picker:

• FP8 Hopper, no calibration wanted → llm-compressor FP8_DYNAMIC (data-free, ~15 min for a 70B on H100).
• W4A16 INT4 (AWQ or GPTQ) with best accuracy → llm-compressor, AWQModifier or GPTQModifier with 256–512 ultrachat samples.
• NVFP4 on Blackwell → ModelOpt NVFP4_DEFAULT_CFG or llm-compressor NVFP4A16 / NVFP4 scheme (v0.10+; current release 0.13.0, and note the project ships parallel maintenance lines — see references/llm-compressor.md).
• MXFP4 MoE for GPT-OSS-style models → use the vendor checkpoint as-is, or ModelOpt MXFP4.
• KV cache FP8 scales for MLA → llm-compressor kv_cache_scheme block with strategy: tensor (per-tensor is stable; per-head is still the experimental path). The MLA multi-turn corruption once flagged here — #38652 — was fixed by PR #37054 and is no longer a reason to avoid FP8 KV on MLA.

Both tools output a HF directory vLLM serves with --quantization compressed-tensors (llm-compressor) or --quantization modelopt / --quantization modelopt_fp4 (ModelOpt).

Speculative decoding drafters

llm-compressor does not train drafters. ModelOpt does: modelopt/torch/speculative/{eagle,dflash,medusa,plugins}/, examples in examples/speculative_decoding/. Recipes: modelopt_recipes/general/speculative_decoding/{eagle3,dflash}.yaml.

Critical constraint: ModelOpt recipes assume BF16 target — not validated with an already-NVFP4 target (base wrapped in torch.no_grad(), so quantized target is theoretically workable but unvalidated). The order is:

1. Train drafter on BF16 target   (ModelOpt, ~4-12h on 8×H100)
2. Export drafter HF dir          (scripts/export_hf_checkpoint.py)
3. PTQ target to NVFP4 or FP8     (ModelOpt or llm-compressor)
4. (Optional) PTQ drafter too     (small, minimal accuracy cost)
5. Serve both in vLLM             (--quantization modelopt_fp4 --speculative-config '{...}')

Medusa / MTP cannot be trained post-hoc — MTP heads are part of the pretraining (DeepSeek V3, Qwen3-Next, GLM-4.5 MoE, etc.). Full details + exact commands: references/modelopt.md § speculative-decoding.

For spec-dec runtime tuning (acceptance rate metrics, method selection per target family, chunked-prefill composability) use the separate vllm-speculative-decoding skill — don't duplicate here.

Online quantization

Introduced by the v0.14 redesign (PR #37776). Quantizes a BF16 checkpoint at load time, no pre-quantization step. Trade-off: peak load memory is BF16 size.

# Per-tensor FP8 (static scales, simplest)
vllm serve meta-llama/Llama-3.1-70B --quantization fp8_per_tensor

# Per-block FP8 (dynamic per-token activation scales)
vllm serve meta-llama/Llama-3.1-70B --quantization fp8_per_block

# Weight-only INT8
vllm serve meta-llama/Llama-3.1-70B --quantization int8_per_channel_weight_only

# Fine-grained: per-layer-kind override via --quantization-config (JSON or dotted keys)
vllm serve meta-llama/Llama-3.1-70B \
  --quantization fp8_per_tensor \
  --quantization-config '{"moe":{"activation":"mxfp8"},"ignore":["lm_head"]}'

The advanced schema is {linear: {weight, activation}, moe: {weight, activation}, ignore: [...]}; linear / moe also accept a bare shorthand string. Names come from QUANT_KEY_NAMES in vllm/config/quantization.py. There is no --quantization-config-file flag and no global_scheme key.

Known gotchas — see #39663 (drops bias weights), #34129 (doesn't split MoE across EP), #19020 / #32029 / #32412 (multiple active RFCs). For any bias-ed or MoE model, prefer a pre-quantized checkpoint.

The operator-pain-point shortlist

Internalize these before debugging accuracy / throughput regressions:

1. --kv-cache-dtype fp8 on MLA models → garbage on multi-turn (#38652) — FIXED, guidance reversed 2026-07-21. Closed 2026-05-15 with "Fixed by #37054", a PR that merged 2026-03-18 and was already recorded elsewhere in this skill as the v0.19 MLA KV-scale fix. The issue tracker lagged the fix by ~2 months, and this skill carried the blanket avoid-on-DeepSeek/GLM/Kimi warning for longer still. On a current release FP8 KV on MLA is usable — measure it instead of excluding it. See references/kv-cache.md items 1 and 5.
2. Gemma 4 FP8-block → logit saturation / repetitive garbage (#39407, #39049). Use non-block FP8 or FP16.
3. NVFP4 on Qwen3-Next / hybrid-attention models silently corrupted output when quantization_config.ignore missed linear_attn layers (#40252, fixed + closed 2026-04-20). The underlying pattern still applies to any new hybrid-attention model: always audit the ignore list when quantizing non-standard architectures.
4. Online FP8 drops bias weights (#39663). Any bias-ed target → use pre-quantized checkpoint.
5. Dynamic FP8 + LoRA-merged model on B200 → non-deterministic degenerate output (#39662). Pin static FP8.
6. SM120 (RTX 5090, 6000 Pro) is not a datacenter NVFP4 MoE target (#35065, #31085) — full kernel set is SM100 / SM103 only. Desktop Blackwell is production only for fp8.

The hardware-/version-gated traps (B300 TRTLLM hang, ModelOpt-vs-compressed-tensors export drift, MXFP4-linear-falls-back-to-BF16, A100 TurboQuant crash, Qwen3.5 v0.18 KV regression, MXFP8+DeepGEMM pre-v0.19 crash) live in the full triage playbook with symptoms → PR → workaround: references/troubleshooting.md.

Version-gate highlights

Full matrix in references/version-gates.md. Load-bearing ones:

• v0.27.1 — current stable, released 2026-08-11. One change: quantized DSpark Markov heads (#50424) — DSparkMarkovHead.markov_w2 (a ParallelLMHead) now accepts and forwards quant_config, so W4A16 markov_w2 weights including weight_scale_2 load through the normal quantization dispatch path. Unquantized behaviour is preserved. A new quantizable surface on a spec-dec drafter; see vllm-speculative-decoding. The container images shipped first: vllm/vllm-openai:v0.27.1 (plus -x86_64/-aarch64/-cu129/-ubuntu2404 variants) were pushed 2026-08-11 10:24-10:42Z, before the GitHub release at 10:47Z — so "no release yet" never implies "no image yet". Air-gap staging mirrors that image; it does not wait on a PyPI wheel.
• v0.27.0 — 2026-08-10. --linear-backend now honored for ModelOpt W4A16 (#50273); FP4 Qutlass for compressed-tensors (#43229); ModelOpt FP8 emulation on SM80 (#50019); TurboQuant KV quant mode (#50533); compressed-tensors checkpoints for DeepSeek-V4 (#41276) and Kimi-K3 (#50500). Breaking environment change: PyTorch 2.13.0 / Triton 3.7.1 / Transformers 5.14.1 (#48155, #49223) — rebuild any custom kernel or plugin image.
• v0.26 — v0.26.0 (2026-07-27). nvfp4_per_token online MoE quantization (#48538); Humming w[2-7]a[4,8] weight-only for compressed-tensors (#46390); CuTe-DSL FlashInfer MXFP4 (#48417); MLA kv_cache_dtype_skip_layers (#47309); ROCm HybridW4A16 linear kernel (#40977).
• v0.25 — v0.25.1 (2026-07-14). Run v0.25.1+, not v0.25.0, if serving NVFP4 on multi-GPU: v0.25.0 and earlier corrupt output via the fused allreduce+RMSNorm+quant path on models with Gemma/Qwen-style RMSNorm (PR #48330 — see vllm-performance-tuning § garbage !!! output). v0.25.0 also removed PagedAttention entirely and made Model Runner V2 the default for all dense models.
• v0.22–v0.24 — NVFP4 fused MoE for DeepSeek-V4 (#42209), NVFP4 Cutlass linear on the batch-invariant path (#39912), padded NVFP4 quant kernel (#42774), ModelOpt W4A16 NVFP4 fused MoE with mixed-precision dispatch (#42566), FlashInfer cutedsl NVFP4 GEMM (#42235) and cute-dsl MXFP8 linear (#46393), FP8 weight layout canonicalized to (K, N) (#44735), fp8_e5m2 KV cache allowed for non-fp8 checkpoints (#45040), W8A8 int-quant scheme-selection regression fixed (#46860), and an actionable error on group-size/TP mismatch (#46230) in place of an obscure failure.
• v0.21 — v0.21.0 (2026-05-15); v0.20.0 stable shipped 2026-04-27, followed by v0.20.1 / v0.20.2. Quantization-layer churn continues — re-verify any v0.19-specific claim on upgrade.
• v0.19 — online MXFP8, CompressedTensorsW8A8Mxfp8, ROCm AWQ Marlin, TurboQuant KV, DeepGemm E8M0 fix for Qwen3.5 FP8 on Blackwell, --calculate-kv-scales deprecation, Gemma 4 quantized MoE, B300 / GB300 fixes.
• v0.18 — FP8 KV in Triton MLA decode, FlashInfer Sparse MLA FP8, ModelOpt MXFP8 MoE, AMD Quark W4A8 MXFP4/FP8, MLA crash with AWQ/GPTQ fix.
• v0.17 — per-head KV scales, SM100 MXFP8 kernels, compressed-tensors as ground-truth, ModelOpt mixed precision, Llama-4 attention quant.
• v0.16 — NVFP4/FP8 on Turing via emulation, TP>4 for FP4 GEMM, ModelOpt MXFP8 dense.
• v0.15 — MXFP4 W4A16 for compressed-tensors MoE, FP4 kernel optimization (+65 % on SM100F via 256-bit loads).
• v0.14 — Online quantization redesign, MXFP4 W4A16 for dense.

What to read next

• references/formats.md — per-format deep dive: kernels, config JSON shapes, min-capability, known caveats.
• references/llm-compressor.md — recipe cookbook: FP8_DYNAMIC / W4A16 / AWQ / NVFP4A16 / KV-cache FP8 / model-free PTQ commands with exact calibration budgets and output layouts.
• references/modelopt.md — ModelOpt PTQ (hf_ptq.py) + speculative-decoding training (EAGLE-3, dflash, MTP constraints) + vLLM loader compatibility.
• references/kernels.md — kernel × format × SM dispatch map (Marlin / CUTLASS / DeepGEMM / FlashInfer / TRTLLM / Qutlass / Machete / Triton / Exllamav2).
• references/kv-cache.md — KV-cache quantization: dtypes, per-token-head scales, attention-backend compatibility, calibration.
• references/troubleshooting.md — symptom → known-issue → fix playbook.
• references/version-gates.md — release-by-release quantization changes, v0.14 → v0.27.

External references

Load source, not paraphrase:

• vLLM docs: FP8 W8A8, Quantized KV Cache, AMD Quark.
• vLLM recipes: index.
• llm-compressor docs: index, NVFP4 W4A4, Qwen3.5 NVFP4 MoE.
• compressed-tensors spec: quant_scheme.py, overview.
• NVIDIA: Introducing NVFP4, NVFP4 KV cache, MoE perf leaps on Blackwell.
• Red Hat: Accelerating LLMs with NVFP4, LLM Compressor 0.9, vLLM FP8 foundational.
• vLLM blog: GPT-OSS on Blackwell, DeepSeek-R1 WideEP on GB200, DeepSeek-V3.2 on GB300.
• AMD: FP8 with Quark for vLLM, MXFP4 Llama3.3 with Quark.

When in doubt, read the vLLM source — vllm/model_executor/layers/quantization/ is the ground truth, and the quantization layer churns fast enough that cached knowledge rots inside a release cycle.