vLLM High-Throughput Inference Engine AI Skill Guide (Claude)
Overview & Engine Architecture
vLLM is an open-source high-throughput LLM serving engine. PagedAttention stores KV cache in non-contiguous blocks (OS-page analogy), cutting fragmentation. Continuous batching schedules at iteration granularity; chunked prefill (default on V1 when possible) interleaves long prefills with decode so TTFT spikes do not starve live streams. Agents size GPU memory utilization vs max model length vs concurrency, expose the OpenAI-compatible HTTP API, and treat preemption warnings as capacity signals—not noise.
┌─────────────────────────────────────────────────────────────┐
│ vLLM engine topology │
│ │
│ Ingress: OpenAI-compatible HTTP │
│ ├── /v1/chat/completions | /v1/completions | /v1/models │
│ ├── /health (+ metrics / Prometheus in prod) │
│ └── SSE token streaming │
│ │
│ Scheduler + PagedAttention │
│ ├── Continuous batching (preempt / recompute when KV full) │
│ ├── Block table: logical tokens → physical KV blocks │
│ └── Chunked prefill + max_num_batched_tokens budget │
│ │
│ Workers │
│ ├── Tensor parallel (NCCL) / optional pipeline parallel │
│ ├── Quantization: awq | gptq | fp8 | bitsandbytes │
│ └── FlashAttention / FlashInfer kernels │
└─────────────────────────────────────────────────────────────┘
When to use / when not to
Use when
- Serving one or few dense LLMs with high concurrency on NVIDIA (or supported) GPUs.
- Drop-in OpenAI API compatibility for apps already on
@openai-api clients.
- Need continuous batching + KV paging beyond naïve
@huggingface-transformers generate.
Do not use when
- No GPU / tiny CPU-only demos → Transformers pipeline or cloud API.
- Training or LoRA research loops →
@pytorch / @huggingface-transformers Trainer.
- Multi-model routing, auth, and quotas are the product → put a gateway in front; vLLM is the engine, not the full platform.
Operational Capabilities & Agent Directives
- Prefer
vllm serve <model> (current CLI) over ad-hoc imports for production processes.
- Size
--gpu-memory-utilization (default 0.92) with headroom for CUDA graphs / NCCL; do not push ≥0.95 on multi-tenant boxes.
- Cap
--max-model-len to the product context you actually serve—unused context burns KV.
- On multi-GPU:
--tensor-parallel-size N with Docker --ipc=host and adequate --shm-size; debug hangs with NCCL_DEBUG=INFO.
- Tune
--max-num-batched-tokens: lower (~2048) favors ITL; higher (>8192) favors throughput/TTFT on small models / large GPUs. Chunked prefill is on by default in V1 when possible.
- Treat preemption logs as KV pressure: raise util / TP, or lower
max_num_seqs / context—not as a silent retry loop.
- Health-gate traffic:
curl -f /health and /v1/models before attaching load balancers.
- Gated Hub weights: pass token via env (
HF_TOKEN)—never commit credentials; local OpenAI clients may use a placeholder api_key only when the server does not auth.
Production Example: serve + streaming client
start_vllm.sh:
#!/usr/bin/env bash
set -euo pipefail
MODEL_NAME="${MODEL_NAME:-Qwen/Qwen2.5-7B-Instruct}"
SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-qwen2.5-7b}"
PORT="${PORT:-8000}"
TP_SIZE="${TP_SIZE:-1}"
GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}"
MAX_MODEL_LEN="${MAX_MODEL_LEN:-8192}"
nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv,noheader
export VLLM_LOGGING_LEVEL="${VLLM_LOGGING_LEVEL:-INFO}"
exec vllm serve "$MODEL_NAME" \
--served-model-name "$SERVED_MODEL_NAME" \
--host 0.0.0.0 \
--port "$PORT" \
--tensor-parallel-size "$TP_SIZE" \
--gpu-memory-utilization "$GPU_MEM_UTIL" \
--max-model-len "$MAX_MODEL_LEN" \
--dtype auto
Streaming client:
"""Async OpenAI-compatible client against a local vLLM server."""
from __future__ import annotations
import asyncio
import os
from openai import AsyncOpenAI
async def stream_chat(prompt: str) -> str:
base = os.environ.get("VLLM_BASE_URL", "http://127.0.0.1:8000/v1")
# Server without auth: placeholder key required by SDK shape only
client = AsyncOpenAI(base_url=base, api_key=os.environ.get("VLLM_API_KEY", "EMPTY"))
model = os.environ.get("VLLM_MODEL", "qwen2.5-7b")
stream = await client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": prompt},
],
temperature=0.2,
max_tokens=512,
stream=True,
)
parts: list[str] = []
async for chunk in stream:
delta = chunk.choices[0].delta.content or ""
print(delta, end="", flush=True)
parts.append(delta)
print()
return "".join(parts)
if __name__ == "__main__":
asyncio.run(stream_chat("Explain PagedAttention in three bullets."))
Recipes:
# FP8 / quantized path when weights support it
vllm serve Qwen/Qwen2.5-7B-Instruct --quantization fp8 --gpu-memory-utilization 0.85
# 4-GPU TP for larger instruct weights
vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4 --max-model-len 16384
curl -f http://127.0.0.1:8000/health
curl -s http://127.0.0.1:8000/v1/models | head
Technical Troubleshooting Matrix
| Signature |
Likely cause |
Fix path |
| CUDA OOM at load |
Weights + KV reserve > util × VRAM |
Lower --gpu-memory-utilization; shorten --max-model-len; quantize; increase TP |
OOM / ncclUnhandledCudaError without OutOfMemoryError |
NCCL buffers after KV pool filled |
Lower blocks override / util; ensure NCCL init accounted; upgrade vLLM/NCCL |
| Preemption warnings / latency spikes |
KV cache thrash under concurrency |
Lower max_num_seqs; raise util/TP; confirm chunked prefill; watch KV metrics |
| High TTFT under load |
Long prefills starving decode |
Keep chunked prefill; tune max_num_batched_tokens |
| Multi-GPU hang / NCCL timeout |
Missing IPC/shm or bad P2P topology |
Docker --ipc=host --shm-size=16g; NCCL_P2P_DISABLE=1 on broken PCIe; matching GPU SKUs |
max-model-len ValueError |
Requested len > model / rope limit |
Align with model card; use documented rope-scaling only when supported |
| Hub 401 on gated model |
Missing HF token |
HF_TOKEN from secret store; accept license |
| Client 404 model name |
--served-model-name ≠ client model |
Align names in /v1/models |
Best Practices
- Log startup line for
# GPU blocks — that is your KV budget.
- Pin model revision / snapshot for reproducibility; record vLLM version in deploy manifests (
@docker / @kubernetes).
- Load-test with realistic prompt/decode lengths before raising concurrency SLOs.
- Separate staging TP/util experiments from prod; change one capacity knob at a time.
- Put auth, quotas, and TLS at the gateway—do not assume open
0.0.0.0 is safe.
Limitations
- Engine quality tracks model + tokenizer chat template parity from training (
@huggingface-transformers).
- Not a training framework; not a multi-model control plane.
- Hardware/driver/NCCL skew causes non-portable flags across fleets.
- Stop and ask if GPU SKU count, target context, or concurrency SLO are unspecified.
Related Skills
@huggingface-transformers — model export, chat templates, fine-tunes
@pytorch — CUDA/AMP fundamentals and NCCL-adjacent debugging context
@openai-api — client patterns against compatible servers
@docker — GPU container flags (ipc, shm-size, device mounts)
@mlflow — record serving config experiments / eval metrics
Agent Operational Directive
MANDATORY: Budget KV with gpu-memory-utilization and max-model-len before promising concurrency. Prefer vllm serve. Health-check before routing traffic. Treat preemption as capacity failure. Never embed Hub or API secrets in scripts—read from the environment. Keep served model name aligned with clients.
Source anchors (research)
1---2name: vllm3description: "Deploy and tune vLLM OpenAI-compatible servers: PagedAttention KV budgeting, continuous batching, chunked prefill, tensor parallel, quantization, and preemption/OOM failure modes."4category: development5risk: safe6source: self7source_type: self8date_added: "2026-09-13"9tags: ["vllm", "llm-inference", "paged-attention", "continuous-batching", "tensor-parallel", "openai-api", "kv-cache"]10tools: ["claude", "cursor", "gemini", "codex"]11metadata:12 source: https://skillmd.com/skills/alivirgo/vllm13---1415# vLLM High-Throughput Inference Engine AI Skill Guide (Claude)1617## Overview & Engine Architecture1819vLLM is an open-source **high-throughput LLM serving engine**. **PagedAttention** stores KV cache in non-contiguous blocks (OS-page analogy), cutting fragmentation. **Continuous batching** schedules at iteration granularity; **chunked prefill** (default on V1 when possible) interleaves long prefills with decode so TTFT spikes do not starve live streams. Agents size **GPU memory utilization vs max model length vs concurrency**, expose the **OpenAI-compatible** HTTP API, and treat preemption warnings as capacity signals—not noise.2021```22┌─────────────────────────────────────────────────────────────┐23│ vLLM engine topology │24│ │25│ Ingress: OpenAI-compatible HTTP │26│ ├── /v1/chat/completions | /v1/completions | /v1/models │27│ ├── /health (+ metrics / Prometheus in prod) │28│ └── SSE token streaming │29│ │30│ Scheduler + PagedAttention │31│ ├── Continuous batching (preempt / recompute when KV full) │32│ ├── Block table: logical tokens → physical KV blocks │33│ └── Chunked prefill + max_num_batched_tokens budget │34│ │35│ Workers │36│ ├── Tensor parallel (NCCL) / optional pipeline parallel │37│ ├── Quantization: awq | gptq | fp8 | bitsandbytes │38│ └── FlashAttention / FlashInfer kernels │39└─────────────────────────────────────────────────────────────┘40```4142---4344## When to use / when not to4546**Use when**4748- Serving one or few dense LLMs with high concurrency on NVIDIA (or supported) GPUs.49- Drop-in OpenAI API compatibility for apps already on `@openai-api` clients.50- Need continuous batching + KV paging beyond naïve `@huggingface-transformers` `generate`.5152**Do not use when**5354- No GPU / tiny CPU-only demos → Transformers pipeline or cloud API.55- Training or LoRA research loops → `@pytorch` / `@huggingface-transformers` Trainer.56- Multi-model routing, auth, and quotas are the product → put a gateway in front; vLLM is the engine, not the full platform.5758---5960## Operational Capabilities & Agent Directives61621. Prefer **`vllm serve <model>`** (current CLI) over ad-hoc imports for production processes.632. Size **`--gpu-memory-utilization`** (default **0.92**) with headroom for CUDA graphs / NCCL; do **not** push ≥0.95 on multi-tenant boxes.643. Cap **`--max-model-len`** to the product context you actually serve—unused context burns KV.654. On multi-GPU: `--tensor-parallel-size N` with Docker `--ipc=host` and adequate `--shm-size`; debug hangs with `NCCL_DEBUG=INFO`.665. Tune **`--max-num-batched-tokens`**: lower (~2048) favors ITL; higher (>8192) favors throughput/TTFT on small models / large GPUs. Chunked prefill is on by default in V1 when possible.676. Treat **preemption** logs as KV pressure: raise util / TP, or lower `max_num_seqs` / context—not as a silent retry loop.687. Health-gate traffic: `curl -f /health` and `/v1/models` before attaching load balancers.698. Gated Hub weights: pass token via env (`HF_TOKEN`)—never commit credentials; local OpenAI clients may use a placeholder api_key only when the server does not auth.7071---7273## Production Example: serve + streaming client7475`start_vllm.sh`:7677```bash78#!/usr/bin/env bash79set -euo pipefail8081MODEL_NAME="${MODEL_NAME:-Qwen/Qwen2.5-7B-Instruct}"82SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-qwen2.5-7b}"83PORT="${PORT:-8000}"84TP_SIZE="${TP_SIZE:-1}"85GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}"86MAX_MODEL_LEN="${MAX_MODEL_LEN:-8192}"8788nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv,noheader89export VLLM_LOGGING_LEVEL="${VLLM_LOGGING_LEVEL:-INFO}"9091exec vllm serve "$MODEL_NAME" \92 --served-model-name "$SERVED_MODEL_NAME" \93 --host 0.0.0.0 \94 --port "$PORT" \95 --tensor-parallel-size "$TP_SIZE" \96 --gpu-memory-utilization "$GPU_MEM_UTIL" \97 --max-model-len "$MAX_MODEL_LEN" \98 --dtype auto99```100101Streaming client:102103```python104"""Async OpenAI-compatible client against a local vLLM server."""105from __future__ import annotations106107import asyncio108import os109110from openai import AsyncOpenAI111112113async def stream_chat(prompt: str) -> str:114 base = os.environ.get("VLLM_BASE_URL", "http://127.0.0.1:8000/v1")115 # Server without auth: placeholder key required by SDK shape only116 client = AsyncOpenAI(base_url=base, api_key=os.environ.get("VLLM_API_KEY", "EMPTY"))117 model = os.environ.get("VLLM_MODEL", "qwen2.5-7b")118119 stream = await client.chat.completions.create(120 model=model,121 messages=[122 {"role": "system", "content": "You are a concise assistant."},123 {"role": "user", "content": prompt},124 ],125 temperature=0.2,126 max_tokens=512,127 stream=True,128 )129 parts: list[str] = []130 async for chunk in stream:131 delta = chunk.choices[0].delta.content or ""132 print(delta, end="", flush=True)133 parts.append(delta)134 print()135 return "".join(parts)136137138if __name__ == "__main__":139 asyncio.run(stream_chat("Explain PagedAttention in three bullets."))140```141142Recipes:143144```bash145# FP8 / quantized path when weights support it146vllm serve Qwen/Qwen2.5-7B-Instruct --quantization fp8 --gpu-memory-utilization 0.85147148# 4-GPU TP for larger instruct weights149vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4 --max-model-len 16384150151curl -f http://127.0.0.1:8000/health152curl -s http://127.0.0.1:8000/v1/models | head153```154155---156157## Technical Troubleshooting Matrix158159| Signature | Likely cause | Fix path |160| :--- | :--- | :--- |161| CUDA OOM at load | Weights + KV reserve > util × VRAM | Lower `--gpu-memory-utilization`; shorten `--max-model-len`; quantize; increase TP |162| OOM / `ncclUnhandledCudaError` without `OutOfMemoryError` | NCCL buffers after KV pool filled | Lower blocks override / util; ensure NCCL init accounted; upgrade vLLM/NCCL |163| Preemption warnings / latency spikes | KV cache thrash under concurrency | Lower `max_num_seqs`; raise util/TP; confirm chunked prefill; watch KV metrics |164| High TTFT under load | Long prefills starving decode | Keep chunked prefill; tune `max_num_batched_tokens` |165| Multi-GPU hang / NCCL timeout | Missing IPC/shm or bad P2P topology | Docker `--ipc=host --shm-size=16g`; `NCCL_P2P_DISABLE=1` on broken PCIe; matching GPU SKUs |166| `max-model-len` ValueError | Requested len > model / rope limit | Align with model card; use documented rope-scaling only when supported |167| Hub 401 on gated model | Missing HF token | `HF_TOKEN` from secret store; accept license |168| Client 404 model name | `--served-model-name` ≠ client `model` | Align names in `/v1/models` |169170---171172## Best Practices1731741. Log startup line for `# GPU blocks` — that is your KV budget.1752. Pin model revision / snapshot for reproducibility; record vLLM version in deploy manifests (`@docker` / `@kubernetes`).1763. Load-test with realistic prompt/decode lengths before raising concurrency SLOs.1774. Separate staging TP/util experiments from prod; change one capacity knob at a time.1785. Put auth, quotas, and TLS at the gateway—do not assume open `0.0.0.0` is safe.179180---181182## Limitations183184- Engine quality tracks model + tokenizer chat template parity from training (`@huggingface-transformers`).185- Not a training framework; not a multi-model control plane.186- Hardware/driver/NCCL skew causes non-portable flags across fleets.187- Stop and ask if GPU SKU count, target context, or concurrency SLO are unspecified.188189---190191## Related Skills192193- `@huggingface-transformers` — model export, chat templates, fine-tunes194- `@pytorch` — CUDA/AMP fundamentals and NCCL-adjacent debugging context195- `@openai-api` — client patterns against compatible servers196- `@docker` — GPU container flags (`ipc`, `shm-size`, device mounts)197- `@mlflow` — record serving config experiments / eval metrics198199---200201## Agent Operational Directive202203> **MANDATORY**: Budget KV with `gpu-memory-utilization` and `max-model-len` before promising concurrency. Prefer `vllm serve`. Health-check before routing traffic. Treat preemption as capacity failure. Never embed Hub or API secrets in scripts—read from the environment. Keep served model name aligned with clients.204205---206207## Source anchors (research)208209- [vLLM `serve` CLI / CacheConfig](https://docs.vllm.ai/en/stable/cli/serve/)210- [Optimization: preemption & chunked prefill](https://docs.vllm.ai/en/latest/configuration/optimization/)211- [Troubleshooting OOM / NCCL notes](https://docs.vllm.ai/en/latest/usage/troubleshooting/)