Dspy Vllm

vLLM — High-Throughput Production Serving for DSPy

Guide the user through serving self-hosted models with vLLM for production DSPy deployments. High concurrency, multi-GPU, OpenAI-compatible API.

Step 1: Understand the setup

Before generating vLLM configuration, clarify:

1. What GPU hardware? — Model (A100, H100, RTX 4090), count, and VRAM per GPU. This determines tensor parallelism and quantization needs.
2. Which model? — Model name and size (7B, 13B, 70B). Determines VRAM requirements and whether quantization is needed.
3. Workload type? — Production serving (concurrent users), batch processing (offline), or optimization (running MIPROv2/BootstrapFewShot)?
4. Already using Ollama locally? — If yes, help them add vLLM for production while keeping Ollama for dev.

What is vLLM

vLLM is a high-throughput inference engine (74k+ GitHub stars) for LLMs. Key features:

• PagedAttention — 4x memory efficiency vs naive attention, serves more concurrent users
• Continuous batching — processes requests as they arrive, no waiting for batch to fill
• Tensor parallelism — split models across multiple GPUs
• OpenAI-compatible API — drop-in replacement, DSPy connects via openai/ provider
• Speculative decoding — use a small draft model to speed up large model generation

When to use vLLM

vLLM requires NVIDIA GPUs (CUDA). It does not support Apple Silicon or AMD GPUs (ROCm support is experimental).

Setup

Install

pip install vllm

Requires: Python 3.9+, NVIDIA GPU with CUDA 12.1+, Linux (recommended) or WSL2.

Start a vLLM server

# Basic — serve a model with OpenAI-compatible API
vllm serve meta-llama/Llama-3.1-8B-Instruct

# With common options
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --dtype auto

The server exposes /v1/chat/completions and /v1/completions endpoints.

Connect DSPy to vLLM

import dspy

lm = dspy.LM(
    "openai/meta-llama/Llama-3.1-8B-Instruct",
    api_base="http://localhost:8000/v1",
    api_key="none",  # required but any value works
    temperature=0.7,
    max_tokens=1000,
)
dspy.configure(lm=lm)

# Now all DSPy modules use your vLLM-served model
classify = dspy.ChainOfThought("text -> category, reasoning")
result = classify(text="Server is down, customers can't log in!")
print(result.category)

Note: dspy.HFClientVLLM is deprecated. Use dspy.LM("openai/...") with api_base instead.

Tensor parallelism (multi-GPU)

Split large models across multiple GPUs:

# 2 GPUs
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 2

# 4 GPUs
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4

Rule of thumb: --tensor-parallel-size = number of GPUs the model needs. A 70B FP16 model needs ~140GB VRAM → 2x A100-80GB or 4x A100-40GB.

GPU sizing guide

Add ~20% overhead for KV cache. --gpu-memory-utilization 0.9 is a good default.

Quantization

Serve quantized models for reduced VRAM:

# AWQ quantized (recommended — fastest)
vllm serve TheBloke/Llama-2-70B-Chat-AWQ \
    --quantization awq

# GPTQ quantized
vllm serve TheBloke/Llama-2-70B-Chat-GPTQ \
    --quantization gptq

AWQ is generally faster than GPTQ on NVIDIA GPUs. Quality loss from INT4 quantization is typically small for 70B+ models.

Key vLLM server options

vllm serve <model> \
    --host 0.0.0.0 \                    # bind address
    --port 8000 \                        # port
    --tensor-parallel-size 1 \           # number of GPUs
    --max-model-len 8192 \               # max sequence length
    --gpu-memory-utilization 0.9 \       # fraction of GPU memory to use
    --dtype auto \                       # auto, float16, bfloat16
    --max-num-seqs 256 \                 # max concurrent sequences
    --enable-prefix-caching \            # cache common prompt prefixes
    --quantization awq \                 # awq, gptq, or none
    --speculative-model <draft-model> \  # enable speculative decoding
    --num-speculative-tokens 5 \         # tokens to speculate
    --pipeline-parallel-size 1 \         # pipeline stages across GPUs (depth)
    --data-parallel-size 1 \             # data parallel replicas
    --api-key <secret>                   # require this key on all requests

Prefix caching

Enable --enable-prefix-caching when many requests share the same system prompt or few-shot prefix (common in DSPy optimized programs):

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --enable-prefix-caching

This caches the KV cache for shared prompt prefixes, significantly speeding up DSPy programs that use the same few-shot demos across requests.

DSPy optimization with vLLM

vLLM handles concurrent requests well, making it faster than Ollama for optimization:

import dspy

lm = dspy.LM(
    "openai/meta-llama/Llama-3.1-8B-Instruct",
    api_base="http://localhost:8000/v1",
    api_key="none",
    max_tokens=1000,
)
dspy.configure(lm=lm)

# MIPROv2 sends many concurrent LM calls — vLLM handles this well
optimizer = dspy.MIPROv2(metric=metric, auto="medium")
optimized = optimizer.compile(program, trainset=trainset)

Tip: Start the vLLM server with --max-num-seqs 256 to handle the optimizer's parallel requests efficiently.

Develop with Ollama, deploy with vLLM

The recommended workflow for self-hosted models:

import os
import dspy

# Same DSPy code, different LM config
if os.environ.get("ENV") == "production":
    # vLLM in production (high throughput, NVIDIA GPU)
    lm = dspy.LM(
        "openai/meta-llama/Llama-3.1-8B-Instruct",
        api_base="http://gpu-server:8000/v1",
        api_key="none",
    )
else:
    # Ollama in development (easy setup, any platform)
    lm = dspy.LM(
        "ollama_chat/llama3.1:8b",
        api_base="http://localhost:11434",
        api_key="",
        num_ctx=8192,
    )

dspy.configure(lm=lm)

# Everything below is identical regardless of backend
program = dspy.ChainOfThought("question -> answer")
program.load("optimized_program.json")
result = program(question="How do refunds work?")

The optimized program (instructions + demos) transfers between backends because DSPy optimizes at the prompt level, not the model level.

Production deployment patterns

Docker

FROM vllm/vllm-openai:latest
ENV MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct
EXPOSE 8000
CMD ["--model", "${MODEL_NAME}", "--host", "0.0.0.0", "--port", "8000", "--max-model-len", "8192"]

Health check

curl http://localhost:8000/health
# Returns 200 when ready

Behind a load balancer

Run multiple vLLM instances behind nginx or a cloud load balancer for horizontal scaling:

# Instance 1 (GPU 0)
CUDA_VISIBLE_DEVICES=0 vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8001

# Instance 2 (GPU 1)
CUDA_VISIBLE_DEVICES=1 vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8002

Gotchas

• Claude uses the deprecated dspy.HFClientVLLM class. This was removed in DSPy 2.5+. Always use dspy.LM("openai/model-name", api_base="http://localhost:8000/v1", api_key="none") instead.
• Claude omits api_key when connecting to vLLM. LiteLLM (which DSPy uses under the hood) requires the api_key parameter even though vLLM does not authenticate. Set api_key="none" — any non-empty string works.
• Claude recommends vLLM for macOS or Apple Silicon users. vLLM requires NVIDIA GPUs with CUDA. If the user mentions macOS, M1/M2/M3/M4, or no NVIDIA GPU, route to /dspy-ollama instead.
• Claude forgets --enable-prefix-caching for DSPy workloads. DSPy optimized programs prepend the same few-shot demos to every request. Without prefix caching, vLLM recomputes the KV cache for those shared tokens on every call. Always recommend it for DSPy serving.
• Claude sets --max-model-len too high for available VRAM. This causes OOM on startup. Calculate available VRAM minus ~20% for KV cache overhead. For a 70B FP16 model on 2x A100-80GB, cap at ~8192 tokens. Suggest --gpu-memory-utilization 0.9 as the default and tell users to lower --max-model-len if they hit OOM.

Additional resources

• vLLM documentation
• vLLM serve CLI reference
• vLLM GitHub
• For API details, see reference.md
• For worked examples, see examples.md

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• LM configuration basics (providers, parameters, caching) — /dspy-lm
• Local development with Ollama — /dspy-ollama
• Deploying as an API (FastAPI wrapper around your DSPy program) — /ai-serving-apis
• Reducing costs (model routing, caching) — /ai-cutting-costs
• Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do