Integration Vllm

vLLM Integration

🎯 Trigger Conditions

Use when asked about serving DSPy programs with vLLM, optimizing inference latency, or high-throughput LLM serving.

📚 Prerequisites

• vllm package installed
• GPU available (recommended)
• Model ready for serving

🛠️ vLLM Integration Patterns

1. Basic vLLM Setup

from vllm import LLM, SamplingParams

# Load model
llm = LLM(model="your-model")

# Configure sampling
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=1024
)

# Generate
outputs = llm.generate("Your prompt", sampling_params)

2. vLLM with DSPy

import dspy
from vllm import LLM

# Create vLLM engine
vllm_engine = LLM(model="your-model")

# Create DSPy LM wrapper
class VLLMLM(dspy.LM):
    def __init__(self, engine):
        super().__init__()
        self.engine = engine
    
    def __call__(self, prompt, **kwargs):
        # Convert prompt to vLLM format
        outputs = self.engine.generate(prompt, **kwargs)
        return [output.text for output in outputs]

# Use with DSPy
vllm_lm = VLLMLM(vllm_engine)
dspy.settings.configure(lm=vllm_lm)

3. Batch Processing

def batch_generate(llm, prompts, batch_size=32):
    all_outputs = []
    for i in range(0, len(prompts), batch_size):
        batch = prompts[i:i+batch_size]
        outputs = llm.generate(batch, sampling_params)
        all_outputs.extend(outputs)
    return all_outputs

4. Performance Optimization

# Configure for high throughput
llm = LLM(
    model="your-model",
    tensor_parallel_size=2,  # Multi-GPU
    max_num_batched_tokens=4096,
    max_num_seqs=256
)

# Use continuous batching
llm = LLM(
    model="your-model",
    enable_chunked_context=True,
    max_num_batched_tokens=4096
)

⚠️ Pitfalls

• Memory usage: vLLM requires significant VRAM
• Model compatibility: Not all models are supported
• Latency: First request has warmup cost
• Batch size: Optimize batch size for your workload

📖 References

• vLLM Documentation
• DSPy vLLM Integration