Vllm Knowledge Patch

vLLM Knowledge Patch

Use this skill when changing, deploying, extending, or debugging vLLM. Start with the quick references below, then load the topic file that matches the work. Treat a project's manifests, runtime help, logs, and tests as the final authority when behavior depends on the installed build or accelerator.

Reference index

Breaking changes and migration checks

Assume V1-only interfaces

• Do not build new integrations around AsyncLLMEngine, LLMEngine, MQLLMEngine, V0 executors, or V0 attention backends. The remaining V0 interfaces and backends were removed after V1 became the default.
• Custom model forward methods must obtain KV-cache and attention metadata through forward_context; the old kv_cache and attn_metadata arguments are gone.
• Recheck extensions that depended on legacy PagedAttention, old tokenizer or chat-template import locations, or legacy api_server.py placement.

Audit removed and renamed settings

• Replace VLLM_ATTENTION_BACKEND with --attention-backend or AttentionConfig/attention_config.
• Replace --convert reward with --convert embed.
• Replace non-GPU-specific metric names carrying a gpu_ prefix and migrate from removed vllm:time_per_output_token_seconds to vllm:inter_token_latency_seconds.
• Remove use of num_lookahead_slots, best_of, LoRA extra vocabulary, swap_space, per-request logits processors, reasoning_content, VLLM_ALL2ALL_BACKEND, BitBlas, Marlin 24, DeepSpeedFp8, and RTN where the deployed version no longer accepts them.
• Do not depend on vLLM mutating CUDA_VISIBLE_DEVICES; pass device_ids for explicit placement. Existing visibility masks change how integer device IDs are interpreted.

Re-pin environment dependencies on upgrade

• Rebuild images and native extensions when the PyTorch, CUDA, Triton, Transformers, compiler, or manylinux baseline changes.
• Ray is no longer guaranteed as a default dependency; install the executor extras explicitly when using Ray.
• Treat cgroup memory and /dev/shm checks as startup requirements in constrained containers.

Defaults that can change output or availability

Make sampling explicit

• V1's seed default became deterministic. Set the seed deliberately when comparing runs, and do not assume caller RNG isolation when V1 multiprocessing is disabled.
• Use top_k=0 to disable top-k sampling; -1 is only a compatibility sentinel on releases that still accept it.
• Set temperature and other sampling values explicitly when model generation_config defaults must not alter behavior after a model change.
• Treat generation_config.max_tokens as a request default, not necessarily a hard ceiling; explicit request values can override it on newer releases.
• Unsupported speculative-decoding sampling parameters can be rejected rather than ignored.

Re-evaluate scheduler and cache defaults

• Async scheduling became default for many supported configurations after an earlier period with corruption hazards. Opt out with --no-async-scheduling when the deployed combination is unsupported.
• Pipeline parallelism is supported with async scheduling on newer releases, but other exclusions can remain hardware- and method-specific.
• V1 enables prefix caching cheaply for ordinary supported models, while hybrid models keep it opt-in and RISC-V forces it off with chunked prefill.
• KV-connector load failures default to failing the request on newer releases; configure recomputation explicitly if that is the intended policy.
• Cascade attention is no longer assumed enabled, and backend defaults differ across Blackwell, SM90+, sparse MLA, and KV-cache formats.

Understand automatic sizing

• --max-model-len auto selects a context length that fits available GPU memory.
• Scheduler token and sequence defaults vary by device class, offline versus server use, and performance mode. Explicit limits take precedence.
• Automatic batched-token limits account for model context, multimodal media, sequence limits, and chunked-prefill support.
• Use startup GPU KV cache size and maximum-concurrency estimates as capacity signals, not promises independent of request shape.

Configuration quick reference

Prefer structured configuration

• Dataclass-backed CLI options accept a whole JSON object or dotted keys, such as --attention-config.flash_attn_version=2.
• Size values accept decimal k/m/g/t and binary K/M/G/T suffixes in the supported batch, scheduling, KV-memory, and prefetch settings.
• Nested YAML configuration is supported; use it for repeatable deployments instead of long unreviewable command lines.
• Speculation shorthands --spec-method, --spec-model, and --spec-tokens populate --speculative-config, but must not duplicate fields already set in the JSON object.

Choose execution topology deliberately

• Prefer pipeline parallelism with tensor_parallel_size=1 when GPU counts do not divide model layers cleanly or the GPUs lack fast interconnects such as NVLink.
• Native multi-node launch requires the multiprocessing DP backend and an nnodes count that evenly divides DP × PP × TP.
• External data-parallel balancing is MoE-only, forces local DP size to one, and requires an explicit or inferred rank.
• Fault tolerance requires external balancing or an explicit DP rank; it is not available with internal balancing.
• Keep Ray cluster traffic on private per-node addresses. Its transport is not an untrusted-network boundary.

Serving quick reference

OpenAI-style endpoints

• The request model field can be optional. Supported surfaces include Chat/Completions, Responses, /score, embeddings, reranking, transcription, batch chat, and extensible endpoints; exact availability is version- and frontend-dependent.
• Completions suffix and Chat image_url.detail are unsupported. Chat's user is accepted but ignored.
• parallel_tool_calls=false limits output to at most one tool call; true permits but cannot force multiple calls.
• Responses can be retrieved and cancelled by response ID. Newer surfaces add structured outputs, reasoning events, prompt embeddings, namespace tools, tool streaming, and chat-template parameters.
• Validate frontend parity before switching between Python and Rust; the Rust path has its own rollout of auth, CORS, pause/resume, TLS, multimodal, and benchmark support.

Operations and observability

• A dead engine returns HTTP 503 from health checks. Use /server_info, /load, /is_sleeping, pause/resume, cache-reset, and tokenizer-info routes only where supported by the selected frontend.
• --enable-log-requests logs identifiers and parameters at INFO and prompt content at DEBUG. Choose VLLM_LOGGING_LEVEL with that disclosure boundary in mind.
• Use --aggregate-engine-logging for aggregate DP statistics and --fail-on-environ-validation to make validation failures fatal.
• Rotate TLS keys through the server facility where supported; configure cipher policy and graceful-shutdown timeouts explicitly.
• Treat the gRPC interface described as private-use as unsuitable for an exposed network even when a later frontend adds TLS or mTLS.

Model, multimodal, and adapter quick reference

• V1 out-of-tree multimodal support uses the merged processor plus the appropriate get_*_embeddings methods. The legacy input mapper is not a future-proof extension point.
• Multimodal preprocessing, processor caches, prefix hashes, and encoder caches are distinct reuse layers; size and secure them independently.
• Model Runner V2 is selected automatically for increasing model families and can fall back when a requested feature is unsupported. Verify the runner in logs before attributing behavior to it.
• LoRA support spans sharded MoE, multimodal towers/connectors, remote resolvers, quantized adapters, audio adapters, and target-module filtering, but each path has model and backend constraints.
• Use LLM.sleep, LLM.wake_up, pause/resume, weight reload/update, and synchronization APIs according to the integration's request-preservation and serialization requirements.

Quantization and speculation quick reference

• Check both the quantization format and accelerator generation. A format name alone does not establish kernel availability, KV-cache support, or LoRA compatibility.
• Per-layer QuantSpec configuration separates linear and moe choices; out-of-tree methods register a QuantizationConfig and return layer-specific method implementations.
• TurboQuant KV cache currently forces FlashAttention 2 when an incompatible version is selected implicitly.
• In speculative decoding, use draft_tensor_parallel_size inside speculative_config; target-model tensor_parallel_size is not a substitute.
• Heterogeneous-vocabulary speculation is draft-model-only and greedy-only. Token distribution can remain lossless while individual log probabilities still vary with precision and batching.
• Cloud-storage model URIs skip automatic speculator detection; configure the method and proposer explicitly.

Debugging order

1. Confirm the installed vLLM build, frontend, runner, accelerator, wheel, and dependency versions.
2. Capture the resolved engine configuration and startup capacity logs.
3. Make sampling, scheduling, attention, cache, and topology settings explicit.
4. Reduce to one frontend and one execution topology before testing adapters, quantization, or speculative decoding.
5. For distributed failures, verify ranks, private addresses, /dev/shm, locked memory, Ray membership, and NCCL transport selection.
6. For output drift, compare generation configuration, seed, parser, chat template, tokenizer, speculative method, and logprob mode.
7. Load the matching reference file before changing a deprecated interface or assuming support for a model, hardware generation, or endpoint.