Model Serving

Model Serving Architecture

Choosing the right inference server is critical for production AI systems. This skill guides the decision between vLLM, TGI, and Ollama based on throughput, latency, and operational requirements.

vLLM

Best for: High-throughput, multi-GPU clusters, latency-sensitive systems.

vLLM is a high-throughput inference engine with:

• Continuous batching: requests don't wait for full batches; served dynamically
• PagedAttention: KV cache block management reduces memory fragmentation by 20-50%
• Quantization support: AWQ, GPTQ (INT4/INT8), FP8, with <1% quality loss at INT4 AWQ
• Distributed inference: tensor parallelism across multiple GPUs
• Dynamic SLO scheduling: prioritize low-latency requests, batch latency-tolerant ones

Use when:

• Throughput >100 RPS or large models (70B+)
• Multi-GPU or multi-node setup available
• SLOs: P95 latency <500ms with high throughput
• Cost is secondary to performance

Example: vllm serve meta-llama/Llama-2-70b --gpu-memory-utilization 0.9 --enable-prefix-caching

TGI (Text-Generation-Inference)

Best for: Mid-scale deployments, streaming responses, simpler operational burden.

TGI from HuggingFace provides:

• Flash Attention 2: faster attention computation (10-30% latency improvement)
• Continuous batching: similar to vLLM but simpler API
• Streaming: native OpenAI-compatible streaming
• Quantization support: bitsandbytes INT8, GPTQ
• Single-GPU deployment: works efficiently on A100 or L40

Use when:

• Single GPU or 2-4 GPU setup
• Streaming responses important (chat, real-time generation)
• Moderate throughput (10-50 RPS)
• Prefer simplicity over maximum optimization

Example: text-generation-launcher --model-id meta-llama/Llama-2-7b --max-batch-size 32 --quantize bitsandbytes

Ollama

Best for: Local development, edge deployment, single-user serving, quick experimentation.

Ollama abstracts model management and serving:

• Automatic quantization: INT4 GGUF models with near-baseline quality
• Model library: ollama pull llama2:13b (pulls pre-quantized models)
• HTTP API: simple REST interface
• Multi-format support: GGUF, GGML, safetensors
• CPU and GPU support: seamless fallback if GPU unavailable

Use when:

• Development or local testing
• Edge deployment (laptop, RPi, Jetson)
• Single-user or small-scale (1-5 QPS)
• Operational simplicity valued over performance

Example: ollama pull mistral:7b → curl http://localhost:11434/api/generate -d '{"model":"mistral:7b","prompt":"Hello"}'

Comparison Table

Serving Decision Tree

┌─ Is this local/edge deployment?
│  ├─ YES → Use Ollama
│  └─ NO ─┐
│         ├─ Do you have >1 GPU or expect >50 RPS?
│         │  ├─ YES → Use vLLM
│         │  └─ NO ─┐
│         │         ├─ Is streaming response required?
│         │         │  ├─ YES → Use TGI
│         │         │  └─ NO → Use vLLM (higher perf) or TGI (simpler)

Decision: Match infrastructure and SLOs to the server's sweet spot. Avoid over-provisioning with vLLM for <10 RPS or under-provisioning with Ollama for >100 RPS.

Quantization Guide

Quantization reduces model size and memory footprint with minimal quality loss.

Baselines

• FP32 (full precision): 4 bytes per param; 70B model = 280 GB
• FP16 (half precision): 2 bytes per param; 70B model = 140 GB; no quality loss

Quantization Levels

Quantization Strategies

1. Start with FP16: establish baseline quality and latency
2. Try INT8 first: 2× compression, <1% loss, supported by most servers
3. INT4 (AWQ) if needed: 4× compression, still <5% loss; supports streaming
4. Avoid sub-INT4: quality cliff (>10% loss) not justified by marginal size savings

Tool Support

• vLLM: --quantization awq or --quantization gptq
• TGI: --quantize bitsandbytes (INT8 only)
• Ollama: automatic; specify mistral:7b-q4_K_M for INT4 GGUF

Batching Patterns

Dynamic Batching (vLLM Default)

Requests enter a pool and are batched dynamically without waiting. Best for throughput.

Time │ Request A    (500 tokens)
     │ Request B  (300 tokens, enters 100ms later)
     │ Request C  (400 tokens, enters 150ms later)
     ↓
     Batch [A, B, C] serves all three simultaneously
     No request waits for a full batch.

Advantage: No artificial latency; P95 latency remains low even under high load.

When to use: High throughput, latency-tolerant workloads (batch inference, backfill).

Continuous Batching (vLLM with PagedAttention)

Similar to dynamic batching but with block-based KV cache management. Reduces fragmentation.

KV cache blocks: [A, B, C] mapped to logical sequences
Can reuse blocks even if sequence lengths change mid-flight.
Result: 20-50% better GPU utilization.

When to use: Always, if using vLLM. It's the default.

Static Batching (Avoid)

Wait for a fixed batch size before serving. Introduces artificial latency.

Batch size = 32
Request 1-31 arrive; wait.
Request 32 arrives.
Serve all 32.
Requests 1-31 waited even though they could be served at request 10.

Never use: Kills tail latency. Only acceptable in batch jobs (nightly reranking), not interactive services.

Scaling Rules

Vertical Scaling (Single GPU, Optimize Before Scaling Out)

1. Increase batch size until GPU memory is 85-90% utilized
2. Enable continuous batching (vLLM): reduces fragmentation by 20-50%
3. Quantize (INT4 AWQ): 4× compression, <5% quality loss
4. Enable prefix caching (vLLM): deduplicate KV blocks across requests with shared prefixes
5. Lower precision (FP16 instead of FP32): 2× memory savings

Metric: Monitor GPU memory %, P99 latency, tokens/sec. Stop when P99 latency degrades or memory hits 95%.

Horizontal Scaling (Multi-GPU or Multi-Node)

1. Before scaling out: use tensor parallelism on a single node (shard model across GPUs)
   • 70B model: tensor parallelism across 2× A100 80GB GPUs
   • 70B model: tensor parallelism across 8× H100 80GB GPUs for <100ms latency
2. When to scale out: tensor parallelism exhausted OR latency SLOs unmet
   • Replicate behind a load balancer (nginx, Kubernetes service)
   • Each replica handles independent requests
3. Monitoring: watch GPU memory % and queue depth per replica
   • If queue depth >10 requests: add replica
   • If GPU utilization <60% AND latency is good: consolidate replicas

KV Cache Monitoring

Monitor KV cache memory pressure to detect when to scale:

available_KV_cache = (GPU_memory_total - model_weights - activations)
cache_pressure = KV_cache_used / available_KV_cache

If cache_pressure > 0.9 for 5 min → add GPU or reduce batch size
If cache_pressure > 0.95 → drop low-priority requests or return error

For streaming: KV cache grows as output tokens are generated. If a request generates 1000+ output tokens, reserve space.

Production Checklist

• Inference server chosen: vLLM (>100 RPS or 70B+), TGI (mid-scale or streaming), Ollama (local/edge)
• Quantization strategy: FP16 baseline, INT8 or INT4 AWQ evaluated and benchmarked
• Batching: dynamic/continuous batching enabled; static batching avoided
• GPU utilization monitored: <85% baseline, 90%+ with quantization, alert at 95%
• KV cache monitoring: cache pressure tracked; alerts at 90% and 95% thresholds
• Vertical scaling complete: batch size, precision, prefix caching tuned before horizontal scaling
• Load balancer configured: multiple replicas behind nginx/K8s service with health checks
• SLOs defined: P95 latency, error rate, availability targets
• Cost baseline: cost per inference, cost per token, monthly spend estimate
• Failover tested: replica failure, GPU OOM, model load timeout all have graceful degradation