vLLM performance tuning
Target: operators deploying models on new hardware, chasing throughput / latency / goodput SLOs, or diagnosing perf regressions. Verified against v0.27.0 (2026-08-10); latest stable is v0.27.1 (2026-08-11), a one-change patch that touches no perf surface. Last freshened 2026-08-11.
Companion skills: vllm-benchmarking (measure), vllm-caching (KV), vllm-nvidia-hardware (GPU/GEMM), vllm-configuration (env vars), vllm-observability (metrics).
Tuning levers (apply by goal, not in fixed order)
Always first — characterize the workload. ISL / OSL / req/s / concurrency / SLO (P95 TTFT, P95 TPOT, P95 ITL). "Goodput" = tok/s/GPU under SLO, not raw tok/s. Everything below is keyed off these numbers.
Parallelism + MoE kernels (biggest single wins):
- Pick parallelism (see
references/moe-and-ep.md) — model-fits-1-GPU → TP=1 + replicas (DP); MoE MLA (DeepSeek/Kimi-K2) → DP-attn + EP; multi-node → TP intra + PP inter OR Wide-EP.
- MoE on a new SKU → run
benchmark_moe.py --tune — generates E=*,N=*,device_name=*.json configs. Without tuned configs vLLM logs "Using default MoE config. Performance might be sub-optimal!" = 20-40% throughput loss.
- Wide-EP (
--enable-expert-parallel --enable-eplb --enable-dbo) for DeepSeek/Qwen3/Kimi-K2 at ≥16 GPUs.
Throughput / batching:
auto_tune.sh (benchmarks/auto_tune/) sweeps max_num_seqs × max_num_batched_tokens.--gpu-memory-utilization — raise from 0.90 toward 0.95 until steady OOM margin, then back off. MoE: cap at 0.85 (all-to-all buffers not in accounting).- Chunked prefill (on by default where supported) — raise
--max-num-batched-tokens if TTFT > SLO; lower if ITL > SLO. Know your starting point: the default is device-gated, not 2048 — vllm serve on a ≥70 GiB non-A100 GPU starts at 8192 / max_num_seqs=1024; only below 70 GiB (or on A100) is it 2048 / 256.
--performance-mode {balanced,interactivity,throughput} — one flag for the whole posture; try it before hand-tuning the batching pair.
Latency / graph + compile:
- CUDA graphs — keep
FULL_AND_PIECEWISE (default); the flag is now --cudagraph-capture-sizes / --max-cudagraph-capture-size, and the generated cap already tracks min(max_num_seqs*2, 512).
--async-scheduling — default-on in recent releases unless using spec-dec / PP / unsupported MM path.- Compile cache — pre-bake
$VLLM_CACHE_ROOT/torch_compile_cache on a representative pod; mount as PVC / bake into OCI layer.
Distributed (last resort, only when the lever above is exhausted):
- NCCL — on well-configured clouds do nothing. Bare-metal IB:
NCCL_IB_HCA, NCCL_IB_GID_INDEX, NCCL_NET_GDR_LEVEL. Never NCCL_CUMEM_ENABLE=0 on GB200.
- PD disagg — reach for only after the above are exhausted and prefill interference is the actual bottleneck.
Triage tree ("why is it slow")
From Red Hat's 5-step triage (2026-03-09):
| Symptom |
Look at |
Common cause |
| TTFT high, queue empty |
compute-bound prefill |
chunked-prefill budget too low, no prefix cache, bad parallelism |
| TTFT high, queue growing |
capacity |
raise replicas, raise max_num_seqs, check preemption rate |
| TPOT high, TTFT fine |
decode-bound |
MoE kernel not tuned, wrong attention backend, async sched off |
| ITL spikes |
CUDA-graph miss |
batch sizes fall outside captured buckets |
| Preemptions climbing |
KV thrashing |
raise --watermark, lower --max-num-seqs, or add replicas (--swap-space is gone — V1 always recomputes) |
num_running < configured concurrency |
scheduler stall |
check async-sched blockers, multimodal path, structured output |
DCGM signals (not GPU_UTIL): DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE. Low tensor-core active on a GEMM-bound workload = memory-bound.
Quick-answer router
MoE tuning + expert parallelism + DeepEP + EPLB + parallelism decision matrix → references/moe-and-ep.md
Scheduler knobs + CUDA graphs + torch.compile + compile cache → references/scheduler-and-compile.md
NCCL / InfiniBand / DCGM + PD disaggregation (Nixl/Mooncake/LMCache) → references/distributed.md
Known regressions + vendor quirks (AMD / Ascend / XPU) → references/regressions.md
Full citation anchors → references/sources.md
Top 10 operator mistakes this skill exists to prevent
Not running benchmark_moe.py on a new GPU SKU. Shipped configs cover common combos (H100, H200, A100, MI300X for Mixtral). B200 / B300 / GB200 / MI325X / Jetson Thor / RTX Pro Blackwell almost always need re-tuning. Symptom: Using default MoE config. Performance might be sub-optimal! in logs. Fix: python benchmarks/kernels/benchmark_moe.py --model <moe-model> --tp-size <N> --enable-expert-parallel --tune --save-dir ./configs, then export VLLM_TUNED_CONFIG_FOLDER=./configs. Expected: 20-40% throughput recovery.
gpu_memory_utilization=0.95 on MoE. All-to-all staging buffers (DeepEP, NVSHMEM) aren't in the memory accountant. OOM at high concurrency. Fix: cap at 0.85 for MoE, 0.92 for dense.
TP not divisible by head count. Model has 32 heads, TP=7 → shape mismatch. Rule: num_heads % TP == 0 AND hidden_size % TP == 0.
DeepSeek-V3.2 at TP=8 on H100/H200/B200/B300. FlashMLA-Sparse only uses 16 heads per rank, padded to 64 → overhead. Fix: DP=8, EP=8, TP=1. (DeepSeek-V3.2 recipe)
Llama-4-Maverick with --enable-expert-parallel. Activation density 0.78% (1/128) — AllToAll overhead exceeds parallelism win. EP hurts 7-12% vs TP-only for Maverick. DeepSeek-R1 (3.13%) and Qwen3-235B (6.25%) benefit from EP. Rule: only enable EP when (experts_per_token / total_experts) > 2%.
MLA model with TP=8. Single KV head, TP duplicates ~84.5 GB KV cache per rank. Fix: DP-attention + EP-MoE splits KV to ~0.125 GB/GPU/request.
--async-scheduling with unsupported path. Structured outputs (fixed #26866), spec-dec (#24799 fixed), PP/struct-out/spec-dec/MM umbrella tracker #27679 closed 2025-12-29 — all sub-PRs merged. The vllm-ascend v0.11.0rc2 precision bug (#4649) is fixed — closed 2026-03-13; upgrade rather than disabling async-sched. Symptom: stall, latency regression, or precision loss. Fix: upgrade; only disable if you've reproduced the issue on your version.
Shipping VLLM_MOE_USE_DEEP_GEMM=1 blindly on H200. Between d83f3f7 and 5a84b76 the DeepGEMM MoE M<128 restriction was removed; H200 DeepSeek-R1 EP at concurrency ≤8 regressed 1.5× TTFT. #28882 closed 2026-04-21 — upgrade to v0.19.1+ or current main. For pre-v0.19.1 deployments, the workaround was VLLM_MOE_USE_DEEP_GEMM=0 + FlashInfer FP8 for low-concurrency decode; re-benchmark after upgrading before removing the override.
Skipping compile cache on K8s. First-pod torch.compile = 5-15 min on large models. Fix: pre-bake $VLLM_CACHE_ROOT/torch_compile_cache on one pod, mount as PVC / OCI layer. Llama-4 specifically needs VLLM_DISABLE_COMPILE_CACHE=1 — stale-cache bug.
NCCL_CUMEM_ENABLE=0 on GB200. Disables multi-node NVLink, forces TCP/IB fallback. Nvidia's rule: "users should not need to tune NCCL environment variables" on modern clouds. GB200 set NCCL_NET_GDR_C2C=1. PR #16992 fixed vLLM's defaults.
Operator cheat sheet
MoE tune on new hardware (canonical recipe)
# Step 1 — run the tuner (uses Ray to parallelize across local GPUs)
python benchmarks/kernels/benchmark_moe.py \
--model deepseek-ai/DeepSeek-V3 \
--tp-size 8 --enable-expert-parallel --dtype fp8_w8a8 \
--tune --save-dir ./moe_configs
# Step 2 — point vLLM at the configs
export VLLM_TUNED_CONFIG_FOLDER=./moe_configs
# Step 3 — serve + verify no "default MoE config" warning in logs
vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 8 \
--enable-expert-parallel --enable-eplb --enable-dbo \
--gpu-memory-utilization 0.85
Parallelism first-pick table
| Model family |
Small-scale |
Large-scale (≥16 GPUs) |
| Dense (Llama, Qwen3-dense) |
TP=N, DP=replicas |
TP=8 intra-node + PP=nodes OR TP=8 + DP=N |
| MoE non-MLA (Mixtral, Qwen3-MoE) |
TP=N, EP off |
TP + EP: EP = E / TP |
| MoE MLA (DeepSeek-V3/R1, Kimi-K2) |
DP=N + EP |
Wide-EP: DP-attn + EP-MoE, --enable-eplb |
| Llama-4-Maverick (0.78% density) |
TP only |
TP only (EP hurts) |
| DeepSeek-V3.2 (FlashMLA-Sparse) |
DP=8, EP=8, TP=1 |
same |
Concurrency crossover (8× MI300X benchmarks): ≤128 concurrent → TP wins, ≥512 → DP wins, 256-512 mixed. (AMD MoE playbook)
Scheduler first-pass by workload
| Scenario |
max_num_batched_tokens |
max_num_seqs |
Other |
| Throughput-heavy (batch decode) |
4096-16384 |
256-512 |
async sched on |
| Latency-heavy (chat) |
1024-2048 |
64-128 |
async sched on, --stream-interval 1 |
| Long-context RAG |
8192-16384 |
32-64 |
--enable-prefix-caching, --long-prefill-token-threshold |
| Wide-EP DeepSeek |
8192 |
256 |
--enable-expert-parallel --enable-eplb --enable-dbo, FULL_AND_PIECEWISE |
Compile-level shorthand
| Flag |
Effect |
-O0 |
No compile, no CUDA graphs (= --enforce-eager) |
-O1 |
Simple compile + PIECEWISE graphs |
-O2 |
default — full compile + FULL_AND_PIECEWISE + fusions (AllReduce+RMSNorm +15%, SP+Async-TP +10%, Attention+Quant FP8 +7%) |
-O3 |
reserved (currently = -O2) |
What changed under you, v0.22.0 → v0.27.0
Six minors of execution-path change. These move the baseline a re-tune is
measured against — re-benchmark across any of these boundaries rather than
comparing to numbers taken before them.
v0.26.0 → v0.27.0 (the flag-surface window — these break configs, not just numbers):
--max-num-partial-prefills / --max-long-partial-prefills removed
(#49244, v0.27.0). A recipe carrying them no longer starts. They were V0
fields the V1 oracle already rejected, so nothing was lost.- First-request compilation stalls are gone. FA4 JIT warmup infrastructure
(#47451) plus runner-owned Triton kernel warmup before the first request
(#49903). A pre-v0.27.0 TTFT baseline that swallowed a cold JIT is not
comparable to one taken after.
- torch 2.13.0 / torchvision 0.28.0 (#48155, upstream-flagged as a breaking
environment change). Both AOT-compile gates are now satisfied by default, so
AOT and the mega-artifact are on unless explicitly disabled.
- Startup validation got stricter and now fails fast instead of dying later:
/dev/shm too small for the shm ring buffer (#48879), cgroup memory limits
respected on all platforms (#49966), incompatible nested runtime overrides
rejected (#49247), DCP topology validation (#49777), better data-parallel
launch validation (#49124). Configs that "worked" by accident will now refuse
to boot — that is the fix, not a regression.
stream_interval is a per-request sampling param (#49754), so one
interactive client no longer forces N=1 server-wide.FusedMoE renamed FusedMoEFactory (#44941) — matters only if you import
it; no operator-facing flag changed.
v0.22.0 → v0.25.1:
- Model Runner V2 became the default execution path, in three steps.
Qwen3 (v0.22.0) → + Llama and Mistral dense models (#43458, v0.23.0) →
all dense models (#44443, v0.25.0). MRv2 also gained a FlashInfer sampler
(#42472), breakable CUDA graphs (#44050), pipeline-parallel bubble
elimination (#42187), and full-CUDA-graph-compatible dynamic speculative
decoding (#45953). A throughput figure measured on a pre-MRv2 build is not
comparable to one taken after the switch for the same model.
- DeepEP v2 replaced v1 (#41183, v0.24.0), with follow-on token-bound and
topk-index fixes (#46404, #46432).
- Async EPLB is on by default (#43219, v0.23.0). v0.24.0 then made
NCCL-based EPLB rejected in combination with async EPLB (#44978) — a
config that used to start will now fail fast.
- Sequence parallelism no longer requires DP (#47070, v0.25.0),
+1.9–5.0% E2E throughput — a lever that was previously unavailable in
non-DP deployments.
CUDA_VISIBLE_DEVICES is no longer set internally; use the new
device_ids argument (#45026, v0.24.0). Device-pinning wrappers need review.- PagedAttention was removed entirely (#47361, v0.25.0).
- The Transformers modeling backend is now as fast as native vLLM
(#47187, v0.25.0) — the historical "always convert to a native
implementation for speed" reflex is worth re-testing.
- Notable kernel wins if hunting single-digit percentages: batch-invariant
Cutlass FP8 +28.9% E2E (#40408, v0.22.0), CutlassFP8 padding
pre-processing +13.5% TTFT (#42651), SM90 CUTLASS FP8 odd-M
swap_ab
180–290% kernel (#44572, v0.24.0), reduce-scatter MoE all-reduce
+3.1–3.2% E2E (#46635, v0.25.0).
Key numbers to memorize
| Metric |
Value |
Default max_num_batched_tokens / max_num_seqs, vllm serve on ≥70 GiB non-A100 GPU |
8192 / 1024 |
| Same, on <70 GiB or any A100 |
2048 / 256 (the PR #10544 figure — small-GPU branch only) |
Default CUDA-graph sizes (--cudagraph-capture-sizes) |
[1,2,4] + range(8,256,8) + range(256,max,16), cap min(max_num_seqs*2, 512) |
| H200 Wide-EP DeepSeek-R1 throughput |
2.2k tok/s/GPU vs ~1.5k baseline (vllm.ai/blog/large-scale-serving) |
| GB200 Wide-EP DeepSeek-R1 |
26.2K TPGS prefill, 10.1K TPGS decode, 3-5× H200 (vllm.ai/blog/dsr1-gb200-part1) |
| MLPerf v5.1 Blackwell Ultra |
5,842 tok/s/GPU offline, 2,907 server (NVIDIA blog) |
| DeepEP dispatch (FP8, Azure H100 IB400) |
45.9 GB/s RDMA, 149.8 GB/s NVLink (Azure blog) |
| Activation density cutoff for EP win |
> 2% (below: TP wins) |
Source policy
All claims cite file:line, release-note PR refs, or issue IDs. Full anchor list + vendor-specific sources in references/sources.md. Compiled 2026-04-18 against v0.19.0; freshened 2026-05-28 (v0.21.0) and 2026-07-21 (v0.25.1). Last freshened 2026-08-11 against v0.27.0, by reading the config dataclasses and arg_utils.py at the tag rather than skimming release notes.
Flags rot silently between minors, and release notes will not tell you. This
pass found four documented knobs that no longer exist — --max-num-partial-prefills
and --max-long-partial-prefills (v0.27.0), --preemption-mode and --swap-space
(removed eleven and five months before this pass, and missed by two prior
freshens) — plus --cuda-graph-sizes renamed to --cudagraph-capture-sizes.
Three prior passes verified issues diligently and never verified the flag
surface. When freshening, read vllm/config/*.py + vllm/engine/arg_utils.py
at the target tag and check each flag this skill names, by name.
Treat a CLOSED issue as unfixed until you read why it closed. This pass found #31475 (MI300X FP8 slower than BF16) and #25538 (preempt/resume thrashing) both closed NOT_PLANNED by the inactivity bot, and #35048 stale-marked and heading the same way — none of them fixed. Only #29539 and #34249 closed against real fixes, and #38971 closed with a usable answer (--moe-backend).
Next refresh when v0.26.x ships, or when the Wide-EP GB200 Part II blog lands (still Part I only as of 2026-07-21, not re-probed this pass).
1---2name: vllm-performance-tuning3description: vLLM performance-tuning operator reference — tuning workflow (baseline → bottleneck → knob → re-bench), fused-MoE kernel autotune (`benchmark_moe.py` generates `E=N,N=M,device_name=X.json` configs), DeepEP all-to-all + expert parallelism + EPLB, CUDA graph modes (FULL_AND_PIECEWISE default), torch.compile AOT + compile cache, scheduler knobs (`--max-num-batched-tokens`, `--max-num-seqs`, `--async-scheduling`), TP/EP/DP/PP decision tree, NCCL/DCGM on H100/H200/B200/GB200, PD disaggregation (Nixl/Mooncake/LMCache), known regressions + vendor quirks (v0.14→0.15.1 MiniMax, MI300X FP8<BF16, DeepGEMM M<128 TTFT).4---56# vLLM performance tuning78Target: operators deploying models on new hardware, chasing throughput / latency / goodput SLOs, or diagnosing perf regressions. Verified against **v0.27.0** (2026-08-10); latest stable is **v0.27.1** (2026-08-11), a one-change patch that touches no perf surface. Last freshened 2026-08-11.910Companion skills: `vllm-benchmarking` (measure), `vllm-caching` (KV), `vllm-nvidia-hardware` (GPU/GEMM), `vllm-configuration` (env vars), `vllm-observability` (metrics).1112## Tuning levers (apply by goal, not in fixed order)1314**Always first — characterize the workload.** ISL / OSL / req/s / concurrency / SLO (P95 TTFT, P95 TPOT, P95 ITL). "Goodput" = tok/s/GPU **under SLO**, not raw tok/s. Everything below is keyed off these numbers.1516**Parallelism + MoE kernels (biggest single wins):**17- **Pick parallelism** (see `references/moe-and-ep.md`) — model-fits-1-GPU → TP=1 + replicas (DP); MoE MLA (DeepSeek/Kimi-K2) → DP-attn + EP; multi-node → TP intra + PP inter OR Wide-EP.18- **MoE on a new SKU → run `benchmark_moe.py --tune`** — generates `E=*,N=*,device_name=*.json` configs. Without tuned configs vLLM logs "Using default MoE config. Performance might be sub-optimal!" = 20-40% throughput loss.19- **Wide-EP** (`--enable-expert-parallel --enable-eplb --enable-dbo`) for DeepSeek/Qwen3/Kimi-K2 at ≥16 GPUs.2021**Throughput / batching:**22- **`auto_tune.sh`** (`benchmarks/auto_tune/`) sweeps `max_num_seqs × max_num_batched_tokens`.23- **`--gpu-memory-utilization`** — raise from 0.90 toward 0.95 until steady OOM margin, then back off. MoE: cap at 0.85 (all-to-all buffers not in accounting).24- **Chunked prefill (on by default where supported)** — raise `--max-num-batched-tokens` if TTFT > SLO; lower if ITL > SLO. **Know your starting point: the default is device-gated, not 2048** — `vllm serve` on a ≥70 GiB non-A100 GPU starts at **8192 / `max_num_seqs`=1024**; only below 70 GiB (or on A100) is it 2048 / 256.25- **`--performance-mode {balanced,interactivity,throughput}`** — one flag for the whole posture; try it before hand-tuning the batching pair.2627**Latency / graph + compile:**28- **CUDA graphs** — keep `FULL_AND_PIECEWISE` (default); the flag is now `--cudagraph-capture-sizes` / `--max-cudagraph-capture-size`, and the generated cap already tracks `min(max_num_seqs*2, 512)`.29- **`--async-scheduling`** — default-on in recent releases unless using spec-dec / PP / unsupported MM path.30- **Compile cache** — pre-bake `$VLLM_CACHE_ROOT/torch_compile_cache` on a representative pod; mount as PVC / bake into OCI layer.3132**Distributed (last resort, only when the lever above is exhausted):**33- **NCCL** — on well-configured clouds do nothing. Bare-metal IB: `NCCL_IB_HCA`, `NCCL_IB_GID_INDEX`, `NCCL_NET_GDR_LEVEL`. Never `NCCL_CUMEM_ENABLE=0` on GB200.34- **PD disagg** — reach for only after the above are exhausted and prefill interference is the actual bottleneck.3536## Triage tree ("why is it slow")3738From Red Hat's 5-step triage ([2026-03-09](https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance)):3940| Symptom | Look at | Common cause |41|---|---|---|42| TTFT high, queue empty | compute-bound prefill | chunked-prefill budget too low, no prefix cache, bad parallelism |43| TTFT high, queue growing | capacity | raise replicas, raise `max_num_seqs`, check preemption rate |44| TPOT high, TTFT fine | decode-bound | MoE kernel not tuned, wrong attention backend, async sched off |45| ITL spikes | CUDA-graph miss | batch sizes fall outside captured buckets |46| Preemptions climbing | KV thrashing | raise `--watermark`, lower `--max-num-seqs`, or add replicas (`--swap-space` is gone — V1 always recomputes) |47| `num_running` < configured concurrency | scheduler stall | check async-sched blockers, multimodal path, structured output |4849**DCGM signals** (not `GPU_UTIL`): `DCGM_FI_PROF_SM_OCCUPANCY`, `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`. Low tensor-core active on a GEMM-bound workload = memory-bound.5051## Quick-answer router5253**MoE tuning + expert parallelism + DeepEP + EPLB + parallelism decision matrix** → `references/moe-and-ep.md`5455**Scheduler knobs + CUDA graphs + torch.compile + compile cache** → `references/scheduler-and-compile.md`5657**NCCL / InfiniBand / DCGM + PD disaggregation (Nixl/Mooncake/LMCache)** → `references/distributed.md`5859**Known regressions + vendor quirks (AMD / Ascend / XPU)** → `references/regressions.md`6061**Full citation anchors** → `references/sources.md`6263## Top 10 operator mistakes this skill exists to prevent64651. **Not running `benchmark_moe.py` on a new GPU SKU.** Shipped configs cover common combos (H100, H200, A100, MI300X for Mixtral). B200 / B300 / GB200 / MI325X / Jetson Thor / RTX Pro Blackwell almost always need re-tuning. Symptom: `Using default MoE config. Performance might be sub-optimal!` in logs. Fix: `python benchmarks/kernels/benchmark_moe.py --model <moe-model> --tp-size <N> --enable-expert-parallel --tune --save-dir ./configs`, then `export VLLM_TUNED_CONFIG_FOLDER=./configs`. Expected: 20-40% throughput recovery.66672. **`gpu_memory_utilization=0.95` on MoE.** All-to-all staging buffers (DeepEP, NVSHMEM) aren't in the memory accountant. OOM at high concurrency. Fix: cap at 0.85 for MoE, 0.92 for dense.68693. **TP not divisible by head count.** Model has 32 heads, TP=7 → shape mismatch. Rule: `num_heads % TP == 0` AND `hidden_size % TP == 0`.70714. **DeepSeek-V3.2 at TP=8 on H100/H200/B200/B300.** FlashMLA-Sparse only uses 16 heads per rank, padded to 64 → overhead. Fix: `DP=8, EP=8, TP=1`. ([DeepSeek-V3.2 recipe](https://docs.vllm.ai/projects/recipes/en/latest/DeepSeek/DeepSeek-V3_2.html))72735. **Llama-4-Maverick with `--enable-expert-parallel`.** Activation density 0.78% (1/128) — AllToAll overhead exceeds parallelism win. **EP hurts 7-12%** vs TP-only for Maverick. DeepSeek-R1 (3.13%) and Qwen3-235B (6.25%) benefit from EP. Rule: only enable EP when `(experts_per_token / total_experts) > 2%`.74756. **MLA model with TP=8.** Single KV head, TP duplicates ~84.5 GB KV cache per rank. Fix: DP-attention + EP-MoE splits KV to ~0.125 GB/GPU/request.76777. **`--async-scheduling` with unsupported path.** Structured outputs (fixed #26866), spec-dec (#24799 fixed), PP/struct-out/spec-dec/MM umbrella tracker #27679 closed 2025-12-29 — all sub-PRs merged. The vllm-ascend v0.11.0rc2 precision bug (#4649) is fixed — **closed 2026-03-13**; upgrade rather than disabling async-sched. Symptom: stall, latency regression, or precision loss. Fix: upgrade; only disable if you've reproduced the issue on your version.78798. **Shipping `VLLM_MOE_USE_DEEP_GEMM=1` blindly on H200.** Between `d83f3f7` and `5a84b76` the DeepGEMM MoE M<128 restriction was removed; H200 DeepSeek-R1 EP at concurrency ≤8 regressed 1.5× TTFT. **[#28882](https://github.com/vllm-project/vllm/issues/28882) closed 2026-04-21** — upgrade to v0.19.1+ or current main. For pre-v0.19.1 deployments, the workaround was `VLLM_MOE_USE_DEEP_GEMM=0` + FlashInfer FP8 for low-concurrency decode; re-benchmark after upgrading before removing the override.80819. **Skipping compile cache on K8s.** First-pod torch.compile = 5-15 min on large models. Fix: pre-bake `$VLLM_CACHE_ROOT/torch_compile_cache` on one pod, mount as PVC / OCI layer. Llama-4 specifically needs `VLLM_DISABLE_COMPILE_CACHE=1` — stale-cache bug.828310. **`NCCL_CUMEM_ENABLE=0` on GB200.** Disables multi-node NVLink, forces TCP/IB fallback. Nvidia's rule: **"users should not need to tune NCCL environment variables"** on modern clouds. GB200 set `NCCL_NET_GDR_C2C=1`. PR [#16992](https://github.com/vllm-project/vllm/pull/16992) fixed vLLM's defaults.8485## Operator cheat sheet8687### MoE tune on new hardware (canonical recipe)8889```bash90# Step 1 — run the tuner (uses Ray to parallelize across local GPUs)91python benchmarks/kernels/benchmark_moe.py \92 --model deepseek-ai/DeepSeek-V3 \93 --tp-size 8 --enable-expert-parallel --dtype fp8_w8a8 \94 --tune --save-dir ./moe_configs9596# Step 2 — point vLLM at the configs97export VLLM_TUNED_CONFIG_FOLDER=./moe_configs9899# Step 3 — serve + verify no "default MoE config" warning in logs100vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 8 \101 --enable-expert-parallel --enable-eplb --enable-dbo \102 --gpu-memory-utilization 0.85103```104105### Parallelism first-pick table106107| Model family | Small-scale | Large-scale (≥16 GPUs) |108|---|---|---|109| Dense (Llama, Qwen3-dense) | TP=N, DP=replicas | TP=8 intra-node + PP=nodes OR TP=8 + DP=N |110| MoE non-MLA (Mixtral, Qwen3-MoE) | TP=N, EP off | TP + EP: EP = E / TP |111| MoE MLA (DeepSeek-V3/R1, Kimi-K2) | DP=N + EP | Wide-EP: DP-attn + EP-MoE, `--enable-eplb` |112| Llama-4-Maverick (0.78% density) | TP only | TP only (EP hurts) |113| DeepSeek-V3.2 (FlashMLA-Sparse) | DP=8, EP=8, TP=1 | same |114115Concurrency crossover (8× MI300X benchmarks): ≤128 concurrent → TP wins, ≥512 → DP wins, 256-512 mixed. ([AMD MoE playbook](https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html))116117### Scheduler first-pass by workload118119| Scenario | `max_num_batched_tokens` | `max_num_seqs` | Other |120|---|---|---|---|121| Throughput-heavy (batch decode) | 4096-16384 | 256-512 | async sched on |122| Latency-heavy (chat) | 1024-2048 | 64-128 | async sched on, `--stream-interval 1` |123| Long-context RAG | 8192-16384 | 32-64 | `--enable-prefix-caching`, `--long-prefill-token-threshold` |124| Wide-EP DeepSeek | 8192 | 256 | `--enable-expert-parallel --enable-eplb --enable-dbo`, `FULL_AND_PIECEWISE` |125126### Compile-level shorthand127128| Flag | Effect |129|---|---|130| `-O0` | No compile, no CUDA graphs (= `--enforce-eager`) |131| `-O1` | Simple compile + PIECEWISE graphs |132| **`-O2`** | **default** — full compile + `FULL_AND_PIECEWISE` + fusions (AllReduce+RMSNorm +15%, SP+Async-TP +10%, Attention+Quant FP8 +7%) |133| `-O3` | reserved (currently = `-O2`) |134135### What changed under you, v0.22.0 → v0.27.0136137Six minors of execution-path change. These move the baseline a re-tune is138measured against — re-benchmark across any of these boundaries rather than139comparing to numbers taken before them.140141**v0.26.0 → v0.27.0 (the flag-surface window — these break configs, not just numbers):**142143- **`--max-num-partial-prefills` / `--max-long-partial-prefills` removed**144 (#49244, v0.27.0). A recipe carrying them no longer starts. They were V0145 fields the V1 oracle already rejected, so nothing was lost.146- **First-request compilation stalls are gone.** FA4 JIT warmup infrastructure147 (#47451) plus runner-owned Triton kernel warmup before the first request148 (#49903). A pre-v0.27.0 TTFT baseline that swallowed a cold JIT is not149 comparable to one taken after.150- **torch 2.13.0 / torchvision 0.28.0** (#48155, upstream-flagged as a breaking151 environment change). Both AOT-compile gates are now satisfied by default, so152 AOT and the mega-artifact are on unless explicitly disabled.153- **Startup validation got stricter** and now fails fast instead of dying later:154 `/dev/shm` too small for the shm ring buffer (#48879), cgroup memory limits155 respected on all platforms (#49966), incompatible nested runtime overrides156 rejected (#49247), DCP topology validation (#49777), better data-parallel157 launch validation (#49124). Configs that "worked" by accident will now refuse158 to boot — that is the fix, not a regression.159- **`stream_interval` is a per-request sampling param** (#49754), so one160 interactive client no longer forces `N=1` server-wide.161- **`FusedMoE` renamed `FusedMoEFactory`** (#44941) — matters only if you import162 it; no operator-facing flag changed.163164**v0.22.0 → v0.25.1:**165166- **Model Runner V2 became the default execution path, in three steps.**167 Qwen3 (v0.22.0) → **+ Llama and Mistral dense models** (#43458, v0.23.0) →168 **all dense models** (#44443, v0.25.0). MRv2 also gained a FlashInfer sampler169 (#42472), breakable CUDA graphs (#44050), pipeline-parallel bubble170 elimination (#42187), and full-CUDA-graph-compatible dynamic speculative171 decoding (#45953). A throughput figure measured on a pre-MRv2 build is not172 comparable to one taken after the switch for the same model.173- **DeepEP v2 replaced v1** (#41183, v0.24.0), with follow-on token-bound and174 topk-index fixes (#46404, #46432).175- **Async EPLB is on by default** (#43219, v0.23.0). v0.24.0 then made176 NCCL-based EPLB **rejected** in combination with async EPLB (#44978) — a177 config that used to start will now fail fast.178- **Sequence parallelism no longer requires DP** (#47070, v0.25.0),179 +1.9–5.0% E2E throughput — a lever that was previously unavailable in180 non-DP deployments.181- **`CUDA_VISIBLE_DEVICES` is no longer set internally**; use the new182 `device_ids` argument (#45026, v0.24.0). Device-pinning wrappers need review.183- **PagedAttention was removed entirely** (#47361, v0.25.0).184- **The Transformers modeling backend is now as fast as native vLLM**185 (#47187, v0.25.0) — the historical "always convert to a native186 implementation for speed" reflex is worth re-testing.187- Notable kernel wins if hunting single-digit percentages: batch-invariant188 Cutlass FP8 **+28.9% E2E** (#40408, v0.22.0), CutlassFP8 padding189 pre-processing **+13.5% TTFT** (#42651), SM90 CUTLASS FP8 odd-M `swap_ab`190 **180–290% kernel** (#44572, v0.24.0), reduce-scatter MoE all-reduce191 +3.1–3.2% E2E (#46635, v0.25.0).192193### Key numbers to memorize194195| Metric | Value |196|---|---|197| Default `max_num_batched_tokens` / `max_num_seqs`, `vllm serve` on ≥70 GiB non-A100 GPU | **8192 / 1024** |198| Same, on <70 GiB or any A100 | 2048 / 256 (the PR #10544 figure — small-GPU branch only) |199| Default CUDA-graph sizes (`--cudagraph-capture-sizes`) | `[1,2,4] + range(8,256,8) + range(256,max,16)`, cap `min(max_num_seqs*2, 512)` |200| H200 Wide-EP DeepSeek-R1 throughput | 2.2k tok/s/GPU vs ~1.5k baseline ([vllm.ai/blog/large-scale-serving](https://vllm.ai/blog/large-scale-serving)) |201| GB200 Wide-EP DeepSeek-R1 | 26.2K TPGS prefill, 10.1K TPGS decode, 3-5× H200 ([vllm.ai/blog/dsr1-gb200-part1](https://vllm.ai/blog/dsr1-gb200-part1)) |202| MLPerf v5.1 Blackwell Ultra | 5,842 tok/s/GPU offline, 2,907 server ([NVIDIA blog](https://developer.nvidia.com/blog/nvidia-blackwell-ultra-sets-new-inference-records-in-mlperf-debut/)) |203| DeepEP dispatch (FP8, Azure H100 IB400) | 45.9 GB/s RDMA, 149.8 GB/s NVLink ([Azure blog](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/achieving-optimal-performance-for-deepseek-expert-parallelism-deepep-on-azure/4414699)) |204| Activation density cutoff for EP win | > 2% (below: TP wins) |205206## Source policy207208All claims cite file:line, release-note PR refs, or issue IDs. Full anchor list + vendor-specific sources in `references/sources.md`. Compiled 2026-04-18 against v0.19.0; freshened 2026-05-28 (v0.21.0) and 2026-07-21 (v0.25.1). **Last freshened 2026-08-11 against v0.27.0**, by reading the config dataclasses and `arg_utils.py` at the tag rather than skimming release notes.209210**Flags rot silently between minors, and release notes will not tell you.** This211pass found four documented knobs that no longer exist — `--max-num-partial-prefills`212and `--max-long-partial-prefills` (v0.27.0), `--preemption-mode` and `--swap-space`213(removed **eleven and five months** before this pass, and missed by two prior214freshens) — plus `--cuda-graph-sizes` renamed to `--cudagraph-capture-sizes`.215Three prior passes verified *issues* diligently and never verified the *flag216surface*. When freshening, read `vllm/config/*.py` + `vllm/engine/arg_utils.py`217at the target tag and check each flag this skill names, by name.218219**Treat a `CLOSED` issue as unfixed until you read why it closed.** This pass found #31475 (MI300X FP8 slower than BF16) and #25538 (preempt/resume thrashing) both closed `NOT_PLANNED` by the inactivity bot, and #35048 stale-marked and heading the same way — none of them fixed. Only #29539 and #34249 closed against real fixes, and #38971 closed with a usable *answer* (`--moe-backend`).220221Next refresh when v0.26.x ships, or when the Wide-EP GB200 Part II blog lands (still Part I only as of 2026-07-21, not re-probed this pass).
air-gapped