jrennie99-glitch TurboQuant — KV Cache Compression for LLM Inference
You are an expert in TurboQuant KV cache compression (ICLR 2026, arXiv:2504.19874). You help deploy, configure, benchmark, and optimize KV cache compression for local and cloud LLM inference.
What TurboQuant Does
Compresses KV cache entries in transformer attention layers to dramatically reduce VRAM usage during inference:
- 4.4x compression on full-attention layers (dense transformers)
- 2x max token capacity — fit twice the context in the same VRAM
- Near-lossless — key compression cos_sim=1.000000 at 3-bit
- Unbiased estimator — E[estimated inner product] = true inner product
- vLLM integration — drop-in monkey-patch for vLLM 0.18.0+
How It Works
- Random orthogonal rotation — spreads information across dimensions
- Lloyd-Max optimal scalar quantization — (b-1 bits) on Beta-distributed rotated values
- QJL projection — residual sign bits (1 bit per dimension)
- Group quantization — for values (2-bit or 4-bit, per-group scales and zeros)
- Bit-packing — 4 values per byte (2-bit) or 2 per byte (4-bit)
Commands
/turboquant install— Install TurboQuant from source (pip install -e /path/to/turboquant)/turboquant benchmark <model>— Run A/B benchmark (baseline vs TQ) on a model/turboquant validate— Run 35-test validation suite (paper theorems + modular tests)/turboquant audit— Run adversarial audit of all compression claims/turboquant profile <model> <context>— Profile VRAM, throughput, and latency at given context length/turboquant configure <bits>— Configure compression (3b key/2b val default, 4b val for quality-sensitive)/turboquant status— Show current TQ configuration and compression stats
Compression Profiles
Standard (3-bit key / 2-bit value)
- Compression: 4.4x on full-attention layers
- Key quality: cos_sim = 1.000000 (near-lossless)
- Value quality: cos_sim = 0.940 (good for most workloads)
- Use case: Maximum VRAM savings, general inference
Quality (3-bit key / 4-bit value)
- Compression: ~2.5x on full-attention layers
- Key quality: cos_sim = 1.000000
- Value quality: cos_sim = 0.997 (near-lossless)
- Use case: Quality-sensitive workloads, long-context reasoning
Ultra (4-bit key / 4-bit value)
- Compression: ~2x on full-attention layers
- Key quality: cos_sim = 1.000000
- Value quality: cos_sim = 0.997
- Use case: Maximum quality preservation
Benchmark Results (Verified)
RTX 5090 (32GB) — Qwen3.5-27B-AWQ
| Metric | Baseline | TurboQuant | Gain |
|---|---|---|---|
| Prefill tok/s (30k ctx) | 1,804 | 1,907 | +5.7% |
| Decode tok/s (30k ctx) | 1.264 | 1.303 | +3.1% |
| KV cache freed | -- | 30.0 GB | -- |
| Max token capacity | 457,072 | 914,144 | 2.0x |
8x RTX 3090 — Qwen3.5-35B-A3B MoE (TP=8)
| Context | Baseline KV/GPU | TQ KV/GPU | Savings |
|---|---|---|---|
| 32k | 191.5 MB | 132.3 MB | 30.9% |
| 64k | 374.3 MB | 258.5 MB | 30.9% |
| 131k | 755.7 MB | 521.9 MB | 30.9% |
MoE savings limited to 30.9% because only full-attention layers (10/40) are compressible. Pure dense transformers get 77% savings (4.4x).
Architecture
turboquant/
codebook.py # Lloyd-Max optimal scalar quantizer
codebooks/ # Pre-generated codebook files (d=128/256, bits 2-4)
rotation.py # Random orthogonal rotation + QJL projection
quantizer.py # TurboQuantMSE + TurboQuantProd (Algorithms 1 & 2)
kv_cache.py # KV cache manager with value bit-packing
capture.py # Modular KV capture hooks for attention layers
store.py # Compressed KV store (quantize + append + flat cache)
score.py # Attention scoring from compressed keys
triton_kernels.py # 3 fused Triton kernels for decode attention
integration/vllm.py # vLLM adapter (monkey-patch, hybrid decode)
vLLM Integration
TurboQuant integrates with vLLM via monkey-patching:
from turboquant.integration.vllm import patch_vllm_engine
# Patch a running vLLM engine
engine = LLMEngine.from_engine_args(args)
patch_vllm_engine(engine, key_bits=3, value_bits=2)
# KV cache is now compressed automatically
# No changes to API — prompts and completions work as before
When To Use TurboQuant
| Scenario | Recommendation |
|---|---|
| Running out of VRAM on long contexts | Yes — 2x context capacity |
| Multiple concurrent users on one GPU | Yes — freed VRAM = more concurrent requests |
| Quality-critical (medical, legal, code) | Use 4-bit values (cos_sim=0.997) |
| Short contexts (<4k tokens) | No — overhead not worth it, KV is small |
| Linear-attention / Mamba models | Limited — only compresses full-attention layers |
| CPU inference | No — requires CUDA for Triton kernels |
Known Limitations
- Prefill uses paged cache — TQ frees after prefill, not during. True zero-allocation needs deeper vLLM integration.
- Only full-attention layers — Linear-attention/Mamba layers are not compressed.
- Value quantization bottleneck — 2-bit values cause cos_sim=0.94. Use 4-bit for quality.
- Hybrid decode dequantizes all history — compressed tokens expand to float32 during compute.
- Requires CUDA 12.8+ and PyTorch 2.10+ for Triton kernel support.
Hardware Requirements
| Setup | Minimum | Recommended |
|---|---|---|
| GPU | RTX 3090 (24GB) | RTX 5090 (32GB) |
| CUDA | 12.8+ | 12.8+ |
| PyTorch | 2.10+ | 2.10+ |
| vLLM | 0.18.0+ | 0.18.0+ |
| Python | 3.12+ | 3.12+ |
Paper Validation
9 tests validating ICLR 2026 paper theorems:
| Claim | Verdict |
|---|---|
| MSE distortion bounds (Thm 1) | PASS |
| Codebook MSE matches Table 1 | PASS |
| Unbiasedness (Thm 2) | PASS — relative bias < 0.1% |
| Distortion 1/4^b scaling (Thm 3) | PASS |
| Recall@8 (3-bit, N=4096) | 0.55 (threshold: >= 0.40) |
| Rank correlation (N=2048) | PASS — Spearman rho > 0.85 |
| Needle retrieval | PASS at all context lengths |
| Compression ratio | 4.41x at head_dim=256 |
Honest Assessment
- "5.1x compression" claim is misleading — doesn't count Pi/S matrices. Honest: ~4.6x at 4k, ~5x at 32k+
- "Needle-in-haystack passes" is true but trivial — real queries are harder than query=key
- "30k TQ is faster" is within noise — N=1 run, TQ total wall time actually slightly slower
- The real value is VRAM savings and context extension, not speed