Vllm Speculative Decoding

vLLM speculative decoding — operator skill

For production vLLM operators deciding which speculative method fits a given model + workload, configuring it correctly, wiring the acceptance metrics into their dashboards, and diagnosing why a deployment isn't seeing the expected speedup.

When spec-dec wins, when it loses

Spec-dec amortises memory-bandwidth-bound decode by letting a cheap proposer guess k tokens that a single target-model forward can verify in parallel.

• Wins at low concurrency (BS 1–8): decode is bandwidth-bound, k verified tokens per target step → 1.5–3× throughput on a well-matched target+drafter. EAGLE-3 on Llama-3.1-8B: +32% TPOT over EAGLE-1 at BS=4 (vLLM v0.11.1+, PR #25916). DFlash on Qwen3-8B: 3.5× at BS=1, 1.6× at BS=32 (PR #36847).
• Hurts at high concurrency (BS ≥ 32): target becomes compute-bound, draft latency is no longer hidden, rejections waste GPU time. Red Hat, Snowflake and the P-EAGLE author all report this. Gate spec-dec to the low-concurrency tier of a disagg or multi-tenant deployment, or disable above a threshold.
• Domain mismatch sinks acceptance: stock EAGLE-3 checkpoints are chat-tuned. Code / agentic / RL-rollout workloads see AL drop from ~3 to ~2. Measure on actual traffic before trusting vendor numbers.

Method selection

Pick once by target-model family and workload shape. Full per-method detail in references/methods.md; MTP in references/mtp.md; EAGLE-3 specifics including P-EAGLE in references/eagle3.md; DFlash in references/dflash.md; Arctic plugin and suffix in references/arctic-inference.md. If training your own EAGLE-3 / DFlash drafter (vs. picking a published one), see references/training-data-recipes.md for the five recipe families surveyed across 466 published checkpoints.

EAGLE-3/DFlash aux-hidden-state support — no longer a hardcoded list.

Through ~v0.21 this was a 14-name allowlist in vllm/config/speculative.py (llama, qwen, minicpm, gpt_oss, hunyuan_vl, hunyuan_v1_dense, afmoe, nemotron_h, deepseek_v2, deepseek_v3, kimi_k2, kimi_k25, minimax_m2, gemma4). At v0.27.0 that list is still gone from the file (re-verified 2026-08-11). Support is now a capability interface: SupportsEagle3 in vllm/model_executor/models/interfaces.py, checked by supports_eagle3(model) in vllm/v1/worker/gpu/spec_decode/eagle/eagle3_utils.py, which raises RuntimeError("Model does not support EAGLE3 interface") when absent. Models declare their own layers via get_eagle3_aux_hidden_state_layers() / set_aux_hidden_state_layers().

So the question "is my model supported?" changed shape: grep the model class for the SupportsEagle3 interface, don't look for a name in a config list. speculative.py now only records which methods consume aux hidden states — eagle3, extract_hidden_states, dflash, dspark.

Canonical --speculative-config shapes

Single source of truth: --speculative-config JSON (or -sc alias, v0.19+). Legacy --speculative-model / --num-speculative-tokens CLI is deprecated.

# EAGLE-3 (the default "give me more tokens" choice in 2026)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{"method":"eagle3","model":"yuhuili/EAGLE3-LLaMA3.1-Instruct-8B","num_speculative_tokens":3}'

# MTP (target model has native heads — model field not set)
vllm serve deepseek-ai/DeepSeek-V3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

# DFlash (Qwen3 on B200)
vllm serve Qwen/Qwen3-8B \
  --attention-backend flash_attn \
  --speculative-config '{"method":"dflash","model":"<dflash-checkpoint>","num_speculative_tokens":15,"parallel_drafting":true}'

# Suffix decoding (agentic / code workloads)
# Requires: pip install arctic-inference
vllm serve <target> \
  --speculative-config '{"method":"suffix","num_speculative_tokens":32}'

# N-gram GPU (v0.18+, model-free, repetitive prompts)
vllm serve <target> \
  --speculative-config '{"method":"ngram_gpu","num_speculative_tokens":5,"prompt_lookup_min":2,"prompt_lookup_max":5}'

# Draft model (own tiny same-family LM, TP must match target)
vllm serve <target> \
  --speculative-config '{"method":"draft_model","model":"<tiny-model>","num_speculative_tokens":4,"parallel_drafting":true}'

num_speculative_tokens tuning guidance in references/methods.md. n_predict interaction for MTP (multiples-of-N rule) in references/mtp.md.

Acceptance-rate metric surface

vLLM V1 emits four spec-dec metrics on /metrics (vllm/v1/spec_decode/metrics.py:154-198). Every production spec-dec deployment should scrape and dashboard all four:

Counters export with _total suffix (prometheus_client convention).

PromQL recipes (from source comments in metrics.py:122-139):

# Acceptance rate
rate(vllm:spec_decode_num_accepted_tokens_total[5m]) /
rate(vllm:spec_decode_num_draft_tokens_total[5m])

# Mean acceptance length (+1 for the bonus target token)
1 + (
  rate(vllm:spec_decode_num_accepted_tokens_total[5m]) /
  rate(vllm:spec_decode_num_drafts_total[5m])
)

# Per-position acceptance — watch the tail falloff
rate(vllm:spec_decode_num_accepted_tokens_per_pos_total[5m]) /
rate(vllm:spec_decode_num_drafts_total[5m])

Expected steady-state acceptance (rough bands — measure actuals):

• EAGLE-3 / MTP: 0.75–0.92
• DFlash: 0.80–0.90
• Draft model (well-matched): 0.70–0.85
• ngram / ngram_gpu: 0.30–0.60 (higher on repetitive prompts)
• Medusa: 0.60–0.75
• Alert if below 0.50 — drafter divergence, tokenizer mismatch, or temperature drift.

Alertmanager templates, Grafana-panel layout, and cross-metric diagnostics (AL-falling-while-KV-growing, acceptance-stable-but-throughput-flat, etc.) in references/metrics.md.

Smoke-check a live endpoint with ${CLAUDE_SKILL_DIR}/scripts/check-spec-decode.sh <base-url>.

Critical version gates

Spec-dec shipped many fixes in 2025-2026 that affect correctness, not just perf. If operating off a build older than these, upgrade before benchmarking.

Critical pitfalls

The "wins/loses" section above covers BS regime and domain mismatch. The method-selection matrix covers TP constraints and the Arctic plugin requirement. The items below are the silent-behaviour and version-specific traps not captured by those.

1. Draft-model TP must equal target TP — hard error at vllm/config/speculative.py:46-51, root cause is torch.compile cache corruption. Medusa and mlp_speculator instead silently force TP=1 (validation at line 731), so a TP=8 target with a Medusa drafter runs the drafter serially. Undocumented.
2. MTP num_speculative_tokens > model's native n_predict must be a multiple of n_predict; vLLM re-runs the MTP layer. Logged warning at speculative.py:531-536 ("may result in lower acceptance rate"). DeepSeek-V3 ships n_predict=1 so asking for 5 runs the layer 5× sequentially.
3. DeepSeek-V3.2 MTP forces enforce_eager=True (speculative.py:397-398). No CUDA graphs → ~10-20% throughput hit. Marked FIXME; recheck on upgrade.
4. DFlash requires --attention-backend flash_attn — Triton and FlashInfer-TRTLLM don't support the non-causal cross-attention path.
5. MTP model-specific method names are deprecated. deepseek_mtp, glm4_moe_mtp, qwen3_next_mtp, etc. all unified under method: "mtp" (PR #25232). Old names still work but log deprecation.
6. Logprob stability is not guaranteed with spec-dec. If downstream eval requires reproducible logprobs, disable spec-dec. Lossless token guarantee holds "up to hardware numerics."
7. Tokenizer mismatch between draft and target destroys acceptance. The draft_model validator does a vocab-size check, but not token-id alignment. Verify with a shared-tokenizer family — unless you are on v0.25.0+ and opt in to TLI. TLI is not automatic: set "use_heterogeneous_vocab": true in --speculative-config, or the equal-vocab check still rejects the pair. It requires method: "draft_model" and draft_sample_method: "greedy" (the default); both are hard ValueErrors. See references/methods.md.
8. Quantization mismatch: drafter in FP16 + target in FP8 is fine; drafter in INT4 + target in FP8 causes AL collapse. Align where possible, or measure before adopting. DSpark exception (v0.27.1+): the Markov head is a separate quantizable surface — DSparkMarkovHead.markov_w2 only honours the checkpoint's quantization from #50424 onward. On ≤ v0.27.0 it silently loads unquantized, so a W4A16 DSpark checkpoint uses more drafter VRAM than its card implies.

Composability (current, verified)

chunked_prefill + spec-dec → supported since v0.11.1 (PR #26263 fix). No mutual exclusivity in code. async_scheduling + spec-dec → default since v0.14.0. Zero-bubble variant in v0.19.0 (#32951). LoRA + spec-dec → EAGLE + LoRA CUDA graph specialisation in v0.11.1 (#28318). Nemotron-H MTP LoRA in v0.16.0 (#32265). Pipeline parallel + spec-dec → MRV2 only, v0.17.0+ (#33960). Not V1 on current engine runner; if pinning PP on non-MRV2, spec-dec is off. FP8 KV + spec-dec → supported. Sparse MLA + MTP full CUDA graphs v0.17.0 (#34457). FP8 MLA KV specific fix #37054. From v0.26.0 the drafter can carry its own kv_cache_dtype inside --speculative-config (#48787); unset means it inherits the target's --kv-cache-dtype. Structured outputs + spec-dec → v0.16.0+ (#33374). Prior: incompatible. Disaggregated serving + spec-dec → v0.17.0+ (#34529). KV-transfer fix #35158 in v0.18.0. Multimodal + spec-dec → v0.11.1 EAGLE/EAGLE3 on Qwen2.5-VL (#22872); text-only drafters auto-disabled for MM when mismatch (#25667); v0.19 multimodal embeddings for spec decode (#36097). Broad but spotty; check target family.

Air-gapped operation

Spec-dec checkpoints (EAGLE-3 heads, MLP speculators, LSTM speculators, DFlash adapters) live on Hugging Face. Treat them like any other HF model per the vllm-configuration skill's air-gap patterns (HF_ENDPOINT mirror, HF_HUB_OFFLINE=1, or VLLM_USE_MODELSCOPE=True). Specific gotchas:

• EAGLE-3 auto-detection fires on "eagle3" substring in model field — renaming a locally-cached dir can break detection; keep the upstream name.
• Arctic Inference speculator checkpoints (Llama-3.1-8B/70B, Llama-3.3-70B, Qwen2.5-32B) also live on HF — mirror before cutover.
• MTP heads ship inside the target model checkpoint — no separate artefact. Air-gap story is the same as the base model.

When spec-dec is the wrong lever

• Workload is embedding / classification / reward-model scoring — no autoregressive decode, spec-dec does nothing.
• Single-token generation (tool-call decision, yes/no classification) — overhead without benefit.
• Prefill-dominated traffic (TTFT SLO, long prompts, short outputs) — spec-dec helps decode only. Chunked prefill and prefix cache matter more.
• Already-fast model (small <1B, or quantised to where decode is compute-bound) — headroom for spec-dec is minimal.

See references/troubleshooting.md for diagnostic flow on low acceptance, silent regressions after upgrade, and metric-interpretation pitfalls.

External references

Last verified: 2026-08-11. Source claims are probed against v0.27.0 (2026-08-10) except the DSpark Markov head, probed against v0.27.1 (2026-08-11, current stable). vLLM source and PRs probed via gh; HF-checkpoint and Arctic-Inference rows carry older stamps — see references/sources.md for the per-row provenance table, classifications, and re-verification recipe. The v0.27.1 container images were pushed 2026-08-11 10:24-10:42Z, before the GitHub release at 10:47Z; pull vllm/vllm-openai:v0.27.1 rather than waiting on a PyPI wheel.

• Main docs: https://docs.vllm.ai/en/latest/features/speculative_decoding/
• Per-method docs: /features/speculative_decoding/{eagle,mtp,draft_model,mlp,n_gram,suffix}/
• P-EAGLE blog (2026-03-13): https://vllm.ai/blog/p-eagle
• Speculators v0.3 (2025-12-13): https://vllm.ai/blog/speculators-v030
• Red Hat EAGLE-3 (Jul 2025): https://developers.redhat.com/articles/2025/07/01/fly-eagle3-fly-faster-inference-vllm-speculative-decoding
• Snowflake Arctic Inference: https://www.snowflake.com/en/engineering-blog/fast-speculative-decoding-vllm-arctic/
• SuffixDecoding (NeurIPS 2025): https://arxiv.org/abs/2411.04975
• EAGLE-3 paper: https://arxiv.org/abs/2503.01840
• DeepSeek V3 (MTP §2.2): https://arxiv.org/abs/2412.19437
• Spec-Bench: https://github.com/hemingkx/Spec-Bench
• SPEED-Bench (NVIDIA): https://huggingface.co/blog/nvidia/speed-bench