Vllm Knowledge Patch

vLLM Knowledge Patch

Use this skill when implementing, upgrading, extending, serving, or operating vLLM. Start with the quick references below, then open the topic file that matches the work. Treat the installed package, its configuration, and observed runtime behavior as authoritative when they differ from this guidance.

Reference index

Breaking changes first

Use V1 interfaces only

V1 is the sole engine. Do not build new integrations on AsyncLLMEngine, LLMEngine, MQLLMEngine, V0 attention backends, V0 executors, or the old V0 configuration surface. Custom model forward methods obtain KV cache and attention metadata through forward_context, not explicit kv_cache and attn_metadata parameters.

Audit removed and renamed options during upgrades

Do not carry forward removed num_lookahead_slots, best_of, LoRA extra vocabulary, swap_space, per-request logits processors, or the tokenizer setter. Replace --convert reward with --convert embed; use --moe-backend and --linear-backend instead of backend environment variables. Use device_ids for placement because the runtime no longer sets CUDA_VISIBLE_DEVICES internally.

Pin request behavior explicitly

A model's generation_config can supply chat-template and sampling defaults. Set temperature, token limits, seed, and related sampling controls explicitly when stable behavior matters. generation_config.max_tokens is only a default, not a hard ceiling; an explicit request may exceed it. top_k=0 disables top-k sampling.

Revisit async scheduling and connector failure policy

Async scheduling evolved from experimental and unsafe in some preemption paths to the normal path, and later gained pipeline-parallel and speculative-decoding combinations. Validate it against the installed release and opt out where needed. KV connector load failures default to failing the request; configure recomputation explicitly if that is the intended recovery policy.

Rebuild environments across dependency jumps

Wheel CUDA targets, PyTorch, Transformers, compilers, and container bases have changed repeatedly. Rebuild images and native extensions rather than reusing an older environment. Ray is no longer a default dependency, Python-only installation is optional, and newer builds require a C++20-compatible compiler.

Engine and configuration quick reference

Understand the V1 execution model

EngineCore isolates scheduling and model execution. Tokenization, multimodal preprocessing, detokenization, and response streaming overlap outside it. Scheduling allocates token counts per request rather than treating prefill and decode as separate phases. Prefix caching is cheap and normally enabled for supported non-hybrid models; hybrid models keep it opt-in.

Configure nested dataclasses safely

Pass a complete JSON object or use dotted flags such as:

vllm serve MODEL --attention-config.flash_attn_version=2

Nested JSON accepts decimal lowercase k/m/g/t and binary uppercase K/M/G/T size suffixes. Do not set the same speculative field through both a component shorthand and --speculative-config.

Let automatic sizing work unless capacity is known

--max-model-len auto finds a context length that fits memory. Default batch limits depend on hardware, offline versus server use, and performance mode; they are also adjusted for context length and multimodal media. Inspect startup KV-capacity and maximum-concurrency logs before overriding limits.

Treat hybrid cache settings as correctness-sensitive

Chunked prefill follows model support. Disabling it on a supported generation model or enabling it on an unsupported pooling model can crash or corrupt output. Mamba and hybrid models use aligned prefix state with:

vllm serve MODEL --enable-prefix-caching --mamba-cache-mode align

Choose attention and compilation deliberately

Use --attention-backend or AttentionConfig, not the retired attention backend environment variable. -O0 through -O3 trade startup cost for runtime performance; selective compilation uses compile_ranges. Hardware defaults can change, including Blackwell MLA/prefill choices and cascade attention, so pin a backend when reproducibility matters.

Serving quick reference

Know compatible-API omissions

Completions suffix and Chat image_url.detail are unsupported. Chat user is accepted but ignored. parallel_tool_calls=false guarantees no more than one call; the default permits multiple calls but does not guarantee them.

Use the Responses lifecycle routes

Create at /v1/responses, retrieve at /v1/responses/{response_id}, and cancel at /v1/responses/{response_id}/cancel. The API supports multi-turn input, reasoning items, tools, structured output, streaming, and request-level cache controls; verify the parser required by the selected model.

Separate preprocessing when useful

Render endpoints and vllm launch render allow prompt preprocessing on a GPU-less service. Engines can also consume async generators of StreamingInput for session-oriented workloads such as speech recognition.

Treat gRPC as private transport

The binary HTTP/2 server and Rust control plane provide health, abort, model, server, and KV-event discovery. Even when TLS or mTLS is available on a frontend, keep interfaces documented as private-use behind trusted network boundaries.

Distributed and cache quick reference

Match parallelism to topology

Use pipeline parallelism with tensor parallel size one for uneven GPU counts or poorly connected GPUs such as hosts without NVLink:

vllm serve MODEL --tensor-parallel-size 1 --pipeline-parallel-size 4

Native multi-node launch requires the multiprocessing data-parallel backend; the node count must evenly divide DP × PP × TP. External data-parallel balancing is MoE-only, requires a rank, and is required for fault tolerance.

Secure Ray and provision shared memory

Ray cluster traffic is unencrypted and can carry unsafe payloads; bind each container to a private per-node VLLM_HOST_IP. For GPUDirect RDMA, provide locked memory and a memory-backed /dev/shm, then use NCCL_DEBUG=TRACE to confirm NET/IB/GDRDMA rather than socket transport.

Choose connector recovery and tiers explicitly

KV paths span CPU LRU offload, LMCache, NIXL, Mooncake, 3FS, FlexKV, object-store and filesystem tiers, hybrid allocators, encoder caches, and peer-to-peer secondary tiers. Check layout compatibility, block sizes, prefill/decode parallelism, per-request policy, and failure behavior together.

Quantization quick reference

Validate method, checkpoint, and accelerator as a unit

Quantization names do not imply universal support. Check GPU generation, CPU/XPU/ROCm/TPU path, linear versus MoE layers, checkpoint encoding, attention backend, KV-cache dtype, and LoRA compatibility. TurboQuant KV cache currently forces FlashAttention 2 when no compatible version is pinned.

Register out-of-tree methods before selection

Decorate a QuantizationConfig subclass with @register_quantization_config("name"), implement its dtype/capability/file and dispatch contracts, import the registration module, then select the name. Return a linear quantization method for LinearBase and a FusedMoEMethodBase implementation for FusedMoE; the two contracts differ.

Speculative decoding quick reference

Use method-specific configuration

Use method="suffix" for draft-free dynamic-depth speculation, method="ngram" for prompt lookup, method="mtp" for compatible assistant checkpoints, and method="draft_model" for an explicit drafter. Custom proposers use method="custom_class" with their fully qualified class in the model field.

Keep drafter and target settings separate

Use draft_tensor_parallel_size and the speculative configuration's max_model_len for the drafter. temperature and top_p remain sampling parameters. Heterogeneous vocabulary support is draft-model-only and greedy; it intersects normalized token strings and limits proposals to shared tokens.

Do not promise identical log probabilities

Rejection sampling preserves the target distribution up to numerical precision, and greedy decoding is checked against non-speculative execution. Token log probabilities can still vary with hardware precision and batch composition.

Working method

1. Inspect the installed vLLM version, model configuration, accelerator, and launch mode.
2. Open the relevant topic reference and identify defaults, removals, and feature constraints that affect that exact setup.
3. Prefer explicit request and engine settings when an upgrade changed a default.
4. Validate startup logs, health, cache capacity, request validation, and one representative generation path before production rollout.
5. For distributed changes, also validate rank arithmetic, network privacy, shared memory, connector failure handling, and graceful shutdown.