How do I install the Vllm Serving skill?

Run npx skillmds@latest add amey-thakur/vllm-serving in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Vllm Serving safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Vllm Serving?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Vllm Serving free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Vllm Serving?

Amey-Thakur (@amey-thakur) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Vllm Serving by amey-thakur — Agent Skill

Name: Vllm Serving
Author: Amey-Thakur

Vllm Serving

Deploy an LLM on vLLM with continuous batching, deliberate VRAM planning, and multi-model hosting that never overcommits the card. Use when serving on vLLM and deciding memory fraction, context length, tensor parallel size, and how many models share a GPU.

Amey-Thakur Updated Sep 17, 2026

File contents

vLLM serving

vLLM hands you high throughput almost for free through PagedAttention and continuous batching, but its defaults assume one model owns the entire GPU. The real decision is memory planning: how much VRAM goes to weights, how much to the KV cache, and whether a second model can fit beside the first. Guess wrong and you either strand throughput or hit an out-of-memory kill in the middle of serving traffic.

Method

Serve through the OpenAI-compatible server for production. Launch vllm serve <model> to expose the HTTP endpoint with continuous batching built in. Reserve the offline LLM class for batch scoring jobs where you hold the whole input set up front and need no always-on server.
Choose --gpu-memory-utilization deliberately. The 0.90 default gives vLLM 90 percent of the card for weights plus KV cache. Drop it to 0.80 when a sidecar process shares the GPU; push toward 0.95 only on a dedicated card where nothing else allocates, or the next allocation crashes the server.
Cap --max-model-len to real traffic. vLLM reserves KV space for the declared context up front. Declaring 128k when your p99 prompt-plus-completion is 8k starves the batch of concurrent slots. Set the length to what requests actually use, not the model's architectural maximum.
Read the KV block count vLLM prints at startup. The log reports GPU KV cache size and a number of blocks. Block count times block size is your true concurrency budget. If it is small, shorten context or quantize before you suspect the scheduler of the shortfall.
Add tensor parallelism only when weights exceed one card. Set --tensor-parallel-size to 2, 4, or 8 for a model too large for a single GPU's VRAM. Leave it at 1 when the model fits, since the per-layer all-reduce adds latency you gain nothing from paying.
Host several small models by partition, not overcommit. Give each model its own process with a fixed --gpu-memory-utilization, and make the fractions sum under 1.0 on a shared card, or route models to separate cards. One vLLM process does not time-slice across unrelated models, so split it up front.
Quantize to buy back concurrency. Load AWQ or GPTQ weights, or run FP8 on Hopper, to shrink the weight footprint and hand the reclaimed bytes to the KV cache. Verify output quality on a held-out set before the change ships.

Checks

Does the startup log show enough KV blocks for your target concurrency?
Does --max-model-len track real traffic rather than the model maximum?
Across co-hosted models, do the memory fractions sum to under 1.0?
With tensor parallelism on, did throughput actually rise, not merely spread?

Boundaries

This covers running and sizing vLLM itself. Setting batch size against a latency SLO and the theory behind KV sizing is inference-serving-optimization; sharding a model across nodes is model-parallelism. For graph-compiled engines or non-vLLM runtimes, see tensorrt-optimization and triton-inference-server.

---name: vllm-servingdescription: Deploy an LLM on vLLM with continuous batching, deliberate VRAM planning, and multi-model hosting that never overcommits the card. Use when serving on vLLM and deciding memory fraction, context length, tensor parallel size, and how many models share a GPU.---# vLLM servingvLLM hands you high throughput almost for free through PagedAttention andcontinuous batching, but its defaults assume one model owns the entire GPU. Thereal decision is memory planning: how much VRAM goes to weights, how much to theKV cache, and whether a second model can fit beside the first. Guess wrong andyou either strand throughput or hit an out-of-memory kill in the middle ofserving traffic.## Method1. **Serve through the OpenAI-compatible server for production.** Launch `vllm serve <model>` to expose the HTTP endpoint with continuous batching built in. Reserve the offline `LLM` class for batch scoring jobs where you hold the whole input set up front and need no always-on server.2. **Choose `--gpu-memory-utilization` deliberately.** The 0.90 default gives vLLM 90 percent of the card for weights plus KV cache. Drop it to 0.80 when a sidecar process shares the GPU; push toward 0.95 only on a dedicated card where nothing else allocates, or the next allocation crashes the server.3. **Cap `--max-model-len` to real traffic.** vLLM reserves KV space for the declared context up front. Declaring 128k when your p99 prompt-plus-completion is 8k starves the batch of concurrent slots. Set the length to what requests actually use, not the model's architectural maximum.4. **Read the KV block count vLLM prints at startup.** The log reports GPU KV cache size and a number of blocks. Block count times block size is your true concurrency budget. If it is small, shorten context or quantize before you suspect the scheduler of the shortfall.5. **Add tensor parallelism only when weights exceed one card.** Set `--tensor-parallel-size` to 2, 4, or 8 for a model too large for a single GPU's VRAM. Leave it at 1 when the model fits, since the per-layer all-reduce adds latency you gain nothing from paying.6. **Host several small models by partition, not overcommit.** Give each model its own process with a fixed `--gpu-memory-utilization`, and make the fractions sum under 1.0 on a shared card, or route models to separate cards. One vLLM process does not time-slice across unrelated models, so split it up front.7. **Quantize to buy back concurrency.** Load AWQ or GPTQ weights, or run FP8 on Hopper, to shrink the weight footprint and hand the reclaimed bytes to the KV cache. Verify output quality on a held-out set before the change ships.## Checks- Does the startup log show enough KV blocks for your target concurrency?- Does `--max-model-len` track real traffic rather than the model maximum?- Across co-hosted models, do the memory fractions sum to under 1.0?- With tensor parallelism on, did throughput actually rise, not merely spread?## BoundariesThis covers running and sizing vLLM itself. Setting batch size against a latencySLO and the theory behind KV sizing is inference-serving-optimization; shardinga model across nodes is model-parallelism. For graph-compiled engines ornon-vLLM runtimes, see tensorrt-optimization and triton-inference-server.

Frequently asked questions