# Integration Vllm

> vLLM serving patterns for DSPy

- Skill: `j33bs/integration-vllm` (Agent Skill)
- Install (CLI): `npx skillmds@latest add j33bs/integration-vllm`
- Raw SKILL.md: https://api.skillmd.com/api/skills/j33bs/integration-vllm/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: j33bs (https://skillmd.com/u/j33bs)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/j33bs/integration-vllm

---


# vLLM Integration

## 🎯 Trigger Conditions
Use when asked about serving DSPy programs with vLLM, optimizing inference latency, or high-throughput LLM serving.

## 📚 Prerequisites
- `vllm` package installed
- GPU available (recommended)
- Model ready for serving

## 🛠️ vLLM Integration Patterns

### 1. Basic vLLM Setup
```python
from vllm import LLM, SamplingParams

# Load model
llm = LLM(model="your-model")

# Configure sampling
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=1024
)

# Generate
outputs = llm.generate("Your prompt", sampling_params)
```

### 2. vLLM with DSPy
```python
import dspy
from vllm import LLM

# Create vLLM engine
vllm_engine = LLM(model="your-model")

# Create DSPy LM wrapper
class VLLMLM(dspy.LM):
    def __init__(self, engine):
        super().__init__()
        self.engine = engine
    
    def __call__(self, prompt, **kwargs):
        # Convert prompt to vLLM format
        outputs = self.engine.generate(prompt, **kwargs)
        return [output.text for output in outputs]

# Use with DSPy
vllm_lm = VLLMLM(vllm_engine)
dspy.settings.configure(lm=vllm_lm)
```

### 3. Batch Processing
```python
def batch_generate(llm, prompts, batch_size=32):
    all_outputs = []
    for i in range(0, len(prompts), batch_size):
        batch = prompts[i:i+batch_size]
        outputs = llm.generate(batch, sampling_params)
        all_outputs.extend(outputs)
    return all_outputs
```

### 4. Performance Optimization
```python
# Configure for high throughput
llm = LLM(
    model="your-model",
    tensor_parallel_size=2,  # Multi-GPU
    max_num_batched_tokens=4096,
    max_num_seqs=256
)

# Use continuous batching
llm = LLM(
    model="your-model",
    enable_chunked_context=True,
    max_num_batched_tokens=4096
)
```

## ⚠️ Pitfalls
- **Memory usage**: vLLM requires significant VRAM
- **Model compatibility**: Not all models are supported
- **Latency**: First request has warmup cost
- **Batch size**: Optimize batch size for your workload

## 📖 References
- [vLLM Documentation](https://docs.vllm.ai/)
- [DSPy vLLM Integration](https://dspy-docs.vercel.app/docs/integration/vllm)