AI LLM Inference

LLMOps – Inference & Optimization – Production Skill Hub

Modern Best Practices: vLLM optimizations (24x throughput), FP8/FP4 quantization (30-50% cost reduction), FlashInfer integration, PagedAttention, continuous batching, and production serving patterns.

This skill provides production-ready operational patterns for optimizing LLM inference performance, cost, and reliability. It centralizes decision rules, optimization strategies, configuration templates, and operational checklists for inference workloads.

No theory. No narrative. Only what Claude can execute.

When to Use This Skill

Claude should activate this skill whenever the user asks for:

• Optimizing LLM inference latency or throughput
• Choosing quantization strategies (FP8/FP4/INT8/INT4)
• Configuring vLLM, TensorRT-LLM, or DeepSpeed inference
• Scaling LLM inference across GPUs (tensor/pipeline parallelism)
• Building high-throughput LLM APIs
• Improving context window performance (KV cache optimization)
• Using speculative decoding for faster generation
• Reducing cost per token
• Profiling and benchmarking inference workloads
• Planning infrastructure capacity
• CPU/edge deployment patterns
• High availability and resilience patterns

Scope Boundaries (Use These Skills for Depth)

• Prompting, tuning, datasets → ai-llm
• RAG pipeline construction → ai-rag
• Deployment, APIs, monitoring → ai-mlops
• Safety, governance → ai-mlops

Quick Reference

Decision Tree: Inference Optimization Strategy

Need to optimize LLM inference: [Optimization Path]
    ├─ Primary constraint: Throughput?
    │   ├─ GPU available? → vLLM (24x gain, continuous batching)
    │   └─ CPU only? → llama.cpp + GGUF (Q4_K_M format)
    │
    ├─ Primary constraint: Cost?
    │   ├─ GPU serving? → FP8/FP4 quantization (30-50% reduction)
    │   └─ Needs quality? → FP8 + calibration (99% accuracy retention)
    │
    ├─ Primary constraint: Latency?
    │   ├─ Can use draft model? → Speculative decoding (2-5x speedup)
    │   └─ Long context? → FlashAttention-2 + PagedAttention
    │
    ├─ Large model (>70B)?
    │   ├─ Multiple GPUs? → Tensor parallelism (NVLink required)
    │   └─ Deep model? → Pipeline parallelism (minimize bubbles)
    │
    └─ Edge deployment?
        └─ CPU + quantization (GGUF Q4_K_M) → Optimized for constrained resources

Modern Best Practices (Current Standards)

vLLM Optimizations

• 24x higher throughput vs HuggingFace Transformers
• Continuous batching with PagedAttention
• FlashInfer library integration (NVIDIA collaboration)
• FP8 attention kernels and FP4 GEMMs
• Advanced kernel fusions (AllReduce + RMSNorm + quantization)

Quantization Performance

• FP8/FP4: 30-50% cost reduction, ~99% accuracy retention
• FP4 on large models: 40-50% performance boost (Qwen3-32B)
• GPTQ optimization: 2-3x faster throughput vs default vLLM config
• BF16 → FP8/INT8: ~30% improvement in cost-per-token

Key Tools

• LLM Compressor (vLLM project) - optimized model serving
• NVIDIA TensorRT Model Optimizer - PTQ framework with HuggingFace export
• vLLM + NVIDIA collaboration - FlashInfer kernels, FP8/FP4 support

Resources (Detailed Operational Guides)

For comprehensive guides on specific topics, see:

Infrastructure & Serving

• Infrastructure Tuning - OS, container, Kubernetes optimization for GPU workloads
• Serving Architectures - Production serving stack patterns
• Resilience & HA Patterns - Multi-region, failover, traffic management

Performance Optimization

• Quantization Patterns - FP8/FP4/INT8/INT4 decision trees and validation
• KV Cache Optimization - PagedAttention, FlashAttention, prefix caching
• Parallelism Patterns - Tensor/pipeline/expert parallelism strategies
• Optimization Strategies - Throughput, cost, memory optimization
• Batching & Scheduling - Continuous batching and throughput patterns

Deployment & Operations

• Edge & CPU Optimization - llama.cpp, GGUF, mobile/browser deployment
• GPU Optimization Checklists - Hardware-specific tuning
• Speculative Decoding Guide - Advanced generation acceleration
• Profiling & Capacity Planning - Benchmarking, SLOs, replica sizing

Templates

Inference Configs

Production-ready configuration templates for leading inference engines:

• vLLM Configuration - Continuous batching, PagedAttention setup
• TensorRT-LLM Configuration - NVIDIA kernel optimizations
• DeepSpeed Inference - PyTorch-friendly inference

Quantization & Compression

Model compression templates for reducing memory and cost:

• GPTQ Quantization - GPU post-training quantization
• AWQ Quantization - Activation-aware weight quantization
• GGUF Format - CPU/edge optimized formats

Serving Pipelines

High-throughput serving architectures:

• LLM API Server - FastAPI + vLLM production setup
• High-Throughput Setup - Multi-replica scaling patterns

Caching & Batching

Performance optimization templates:

• Prefix Caching - KV cache reuse strategies
• Batching Configuration - Continuous batching tuning

Benchmarking

Performance measurement and validation:

• Latency & Throughput Testing - Load testing framework

Navigation

Resources

• resources/serving-architectures.md
• resources/profiling-and-capacity-planning.md
• resources/gpu-optimization-checklists.md
• resources/speculative-decoding-guide.md
• resources/resilience-ha-patterns.md
• resources/optimization-strategies.md
• resources/kv-cache-optimization.md
• resources/batching-and-scheduling.md
• resources/quantization-patterns.md
• resources/parallelism-patterns.md
• resources/edge-cpu-optimization.md
• resources/infrastructure-tuning.md

Templates

• templates/serving/template-llm-api.md
• templates/serving/template-high-throughput-setup.md
• templates/inference/template-vllm-config.md
• templates/inference/template-tensorrtllm-config.md
• templates/inference/template-deepspeed-inference.md
• templates/quantization/template-awq.md
• templates/quantization/template-gptq.md
• templates/quantization/template-gguf.md
• templates/batching/template-batching-config.md
• templates/caching/template-prefix-caching.md
• templates/benchmarking/template-latency-throughput-test.md

Data

• data/sources.json — Curated external references

Related Skills

This skill focuses on inference-time performance. For related workflows:

• ai-llm - Prompting, fine-tuning, datasets
• ai-llm - LLM application architecture, RAG pipelines
• ai-rag - RAG pipeline construction and optimization
• ai-mlops - Deployment, APIs, monitoring
• ai-mlops - Safety, governance, security
• qa-observability - Performance monitoring and optimization
• ops-devops-platform - Infrastructure and platform operations

External Resources

See data/sources.json for:

• Serving frameworks (vLLM, TensorRT-LLM, DeepSpeed-MII)
• Quantization libraries (GPTQ, AWQ, bitsandbytes, LLM Compressor)
• FlashAttention, FlashInfer, xFormers
• GPU hardware guides and optimization docs
• Benchmarking frameworks and tools

Use this skill whenever the user needs LLM inference performance, cost reduction, or serving architecture guidance.