# Vllm Knowledge Patch > vLLM - Skill: `nevaberry/vllm-knowledge-patch-2` (Agent Skill, multi-file: 8 files) - Install (CLI): `npx skillmds@latest add nevaberry/vllm-knowledge-patch-2` - Raw SKILL.md: https://api.skillmd.com/api/skills/nevaberry/vllm-knowledge-patch-2/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: nevaberry (https://skillmd.com/u/nevaberry) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/nevaberry/vllm-knowledge-patch-2 --- # vLLM Knowledge Patch Use this skill when changing, deploying, extending, or debugging vLLM. Start with the quick references below, then load the topic file that matches the work. Treat a project's manifests, runtime help, logs, and tests as the final authority when behavior depends on the installed build or accelerator. ## Reference index | Reference | Topics | | --- | --- | | [Engine, Runtime, and Configuration](references/engine-runtime.md) | V1 architecture, scheduling, compilation, batching, configuration, sampling, Model Runner V2, dependencies, and removals | | [Serving APIs, Operations, and Security](references/serving-apis.md) | HTTP, Responses, Chat, embeddings, speech, parsers, Rust frontend, gRPC, metrics, logging, TLS, validation, and security | | [Distributed Execution, Parallelism, and Cache Topology](references/distributed-cache.md) | TP/PP/DP/EP, Ray and multiprocessing, disaggregation, KV connectors, offloading, fault tolerance, and RDMA | | [Models, Multimodal Integration, Adapters, and Lifecycle](references/models-adapters.md) | Model and task coverage, multimodal processing, loading, LoRA, pooling, lifecycle, and RL integrations | | [Quantization Formats, Hardware, and Extensions](references/quantization-hardware.md) | Checkpoint formats, KV-cache quantization, hardware support, custom quantization plugins, wheels, and accelerators | | [Speculative Decoding](references/speculative-decoding.md) | Methods, configuration, acceptance, heterogeneous vocabularies, structured output, and evolving execution combinations | ## Breaking changes and migration checks ### Assume V1-only interfaces - Do not build new integrations around `AsyncLLMEngine`, `LLMEngine`, `MQLLMEngine`, V0 executors, or V0 attention backends. The remaining V0 interfaces and backends were removed after V1 became the default. - Custom model `forward` methods must obtain KV-cache and attention metadata through `forward_context`; the old `kv_cache` and `attn_metadata` arguments are gone. - Recheck extensions that depended on legacy PagedAttention, old tokenizer or chat-template import locations, or legacy `api_server.py` placement. ### Audit removed and renamed settings - Replace `VLLM_ATTENTION_BACKEND` with `--attention-backend` or `AttentionConfig`/`attention_config`. - Replace `--convert reward` with `--convert embed`. - Replace non-GPU-specific metric names carrying a `gpu_` prefix and migrate from removed `vllm:time_per_output_token_seconds` to `vllm:inter_token_latency_seconds`. - Remove use of `num_lookahead_slots`, `best_of`, LoRA extra vocabulary, `swap_space`, per-request logits processors, `reasoning_content`, `VLLM_ALL2ALL_BACKEND`, BitBlas, Marlin 24, DeepSpeedFp8, and RTN where the deployed version no longer accepts them. - Do not depend on vLLM mutating `CUDA_VISIBLE_DEVICES`; pass `device_ids` for explicit placement. Existing visibility masks change how integer device IDs are interpreted. ### Re-pin environment dependencies on upgrade - Rebuild images and native extensions when the PyTorch, CUDA, Triton, Transformers, compiler, or manylinux baseline changes. - Ray is no longer guaranteed as a default dependency; install the executor extras explicitly when using Ray. - Treat cgroup memory and `/dev/shm` checks as startup requirements in constrained containers. ## Defaults that can change output or availability ### Make sampling explicit - V1's seed default became deterministic. Set the seed deliberately when comparing runs, and do not assume caller RNG isolation when V1 multiprocessing is disabled. - Use `top_k=0` to disable top-k sampling; `-1` is only a compatibility sentinel on releases that still accept it. - Set temperature and other sampling values explicitly when model `generation_config` defaults must not alter behavior after a model change. - Treat `generation_config.max_tokens` as a request default, not necessarily a hard ceiling; explicit request values can override it on newer releases. - Unsupported speculative-decoding sampling parameters can be rejected rather than ignored. ### Re-evaluate scheduler and cache defaults - Async scheduling became default for many supported configurations after an earlier period with corruption hazards. Opt out with `--no-async-scheduling` when the deployed combination is unsupported. - Pipeline parallelism is supported with async scheduling on newer releases, but other exclusions can remain hardware- and method-specific. - V1 enables prefix caching cheaply for ordinary supported models, while hybrid models keep it opt-in and RISC-V forces it off with chunked prefill. - KV-connector load failures default to failing the request on newer releases; configure recomputation explicitly if that is the intended policy. - Cascade attention is no longer assumed enabled, and backend defaults differ across Blackwell, SM90+, sparse MLA, and KV-cache formats. ### Understand automatic sizing - `--max-model-len auto` selects a context length that fits available GPU memory. - Scheduler token and sequence defaults vary by device class, offline versus server use, and performance mode. Explicit limits take precedence. - Automatic batched-token limits account for model context, multimodal media, sequence limits, and chunked-prefill support. - Use startup `GPU KV cache size` and maximum-concurrency estimates as capacity signals, not promises independent of request shape. ## Configuration quick reference ### Prefer structured configuration - Dataclass-backed CLI options accept a whole JSON object or dotted keys, such as `--attention-config.flash_attn_version=2`. - Size values accept decimal `k/m/g/t` and binary `K/M/G/T` suffixes in the supported batch, scheduling, KV-memory, and prefetch settings. - Nested YAML configuration is supported; use it for repeatable deployments instead of long unreviewable command lines. - Speculation shorthands `--spec-method`, `--spec-model`, and `--spec-tokens` populate `--speculative-config`, but must not duplicate fields already set in the JSON object. ### Choose execution topology deliberately - Prefer pipeline parallelism with `tensor_parallel_size=1` when GPU counts do not divide model layers cleanly or the GPUs lack fast interconnects such as NVLink. - Native multi-node launch requires the multiprocessing DP backend and an `nnodes` count that evenly divides `DP × PP × TP`. - External data-parallel balancing is MoE-only, forces local DP size to one, and requires an explicit or inferred rank. - Fault tolerance requires external balancing or an explicit DP rank; it is not available with internal balancing. - Keep Ray cluster traffic on private per-node addresses. Its transport is not an untrusted-network boundary. ## Serving quick reference ### OpenAI-style endpoints - The request `model` field can be optional. Supported surfaces include Chat/Completions, Responses, `/score`, embeddings, reranking, transcription, batch chat, and extensible endpoints; exact availability is version- and frontend-dependent. - Completions `suffix` and Chat `image_url.detail` are unsupported. Chat's `user` is accepted but ignored. - `parallel_tool_calls=false` limits output to at most one tool call; `true` permits but cannot force multiple calls. - Responses can be retrieved and cancelled by response ID. Newer surfaces add structured outputs, reasoning events, prompt embeddings, namespace tools, tool streaming, and chat-template parameters. - Validate frontend parity before switching between Python and Rust; the Rust path has its own rollout of auth, CORS, pause/resume, TLS, multimodal, and benchmark support. ### Operations and observability - A dead engine returns HTTP 503 from health checks. Use `/server_info`, `/load`, `/is_sleeping`, pause/resume, cache-reset, and tokenizer-info routes only where supported by the selected frontend. - `--enable-log-requests` logs identifiers and parameters at INFO and prompt content at DEBUG. Choose `VLLM_LOGGING_LEVEL` with that disclosure boundary in mind. - Use `--aggregate-engine-logging` for aggregate DP statistics and `--fail-on-environ-validation` to make validation failures fatal. - Rotate TLS keys through the server facility where supported; configure cipher policy and graceful-shutdown timeouts explicitly. - Treat the gRPC interface described as private-use as unsuitable for an exposed network even when a later frontend adds TLS or mTLS. ## Model, multimodal, and adapter quick reference - V1 out-of-tree multimodal support uses the merged processor plus the appropriate `get_*_embeddings` methods. The legacy input mapper is not a future-proof extension point. - Multimodal preprocessing, processor caches, prefix hashes, and encoder caches are distinct reuse layers; size and secure them independently. - Model Runner V2 is selected automatically for increasing model families and can fall back when a requested feature is unsupported. Verify the runner in logs before attributing behavior to it. - LoRA support spans sharded MoE, multimodal towers/connectors, remote resolvers, quantized adapters, audio adapters, and target-module filtering, but each path has model and backend constraints. - Use `LLM.sleep`, `LLM.wake_up`, pause/resume, weight reload/update, and synchronization APIs according to the integration's request-preservation and serialization requirements. ## Quantization and speculation quick reference - Check both the quantization format and accelerator generation. A format name alone does not establish kernel availability, KV-cache support, or LoRA compatibility. - Per-layer `QuantSpec` configuration separates `linear` and `moe` choices; out-of-tree methods register a `QuantizationConfig` and return layer-specific method implementations. - TurboQuant KV cache currently forces FlashAttention 2 when an incompatible version is selected implicitly. - In speculative decoding, use `draft_tensor_parallel_size` inside `speculative_config`; target-model `tensor_parallel_size` is not a substitute. - Heterogeneous-vocabulary speculation is draft-model-only and greedy-only. Token distribution can remain lossless while individual log probabilities still vary with precision and batching. - Cloud-storage model URIs skip automatic speculator detection; configure the method and proposer explicitly. ## Debugging order 1. Confirm the installed vLLM build, frontend, runner, accelerator, wheel, and dependency versions. 2. Capture the resolved engine configuration and startup capacity logs. 3. Make sampling, scheduling, attention, cache, and topology settings explicit. 4. Reduce to one frontend and one execution topology before testing adapters, quantization, or speculative decoding. 5. For distributed failures, verify ranks, private addresses, `/dev/shm`, locked memory, Ray membership, and NCCL transport selection. 6. For output drift, compare generation configuration, seed, parser, chat template, tokenizer, speculative method, and logprob mode. 7. Load the matching reference file before changing a deprecated interface or assuming support for a model, hardware generation, or endpoint.