# Turboquant

> KV cache compression for LLM inference — 4.4x compression, 2x context capacity, near-lossless quality. ICLR 2026 paper implementation with vLLM integration.

- Skill: `jrennie99-glitch/turboquant` (Agent Skill)
- Install (CLI): `npx skillmds add jrennie99-glitch/turboquant`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jrennie99-glitch/turboquant/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: jrennie99-glitch (https://skillmd.com/u/jrennie99-glitch)
- Updated: 2026-08-19
- Page: https://skillmd.com/skills/jrennie99-glitch/turboquant

---


# TurboQuant — KV Cache Compression for LLM Inference

You are an expert in TurboQuant KV cache compression (ICLR 2026, arXiv:2504.19874). You help deploy, configure, benchmark, and optimize KV cache compression for local and cloud LLM inference.

## What TurboQuant Does

Compresses KV cache entries in transformer attention layers to dramatically reduce VRAM usage during inference:

- **4.4x compression** on full-attention layers (dense transformers)
- **2x max token capacity** — fit twice the context in the same VRAM
- **Near-lossless** — key compression cos_sim=1.000000 at 3-bit
- **Unbiased estimator** — E[estimated inner product] = true inner product
- **vLLM integration** — drop-in monkey-patch for vLLM 0.18.0+

## How It Works

1. **Random orthogonal rotation** — spreads information across dimensions
2. **Lloyd-Max optimal scalar quantization** — (b-1 bits) on Beta-distributed rotated values
3. **QJL projection** — residual sign bits (1 bit per dimension)
4. **Group quantization** — for values (2-bit or 4-bit, per-group scales and zeros)
5. **Bit-packing** — 4 values per byte (2-bit) or 2 per byte (4-bit)

## Commands

- `/turboquant install` — Install TurboQuant from source (`pip install -e /path/to/turboquant`)
- `/turboquant benchmark <model>` — Run A/B benchmark (baseline vs TQ) on a model
- `/turboquant validate` — Run 35-test validation suite (paper theorems + modular tests)
- `/turboquant audit` — Run adversarial audit of all compression claims
- `/turboquant profile <model> <context>` — Profile VRAM, throughput, and latency at given context length
- `/turboquant configure <bits>` — Configure compression (3b key/2b val default, 4b val for quality-sensitive)
- `/turboquant status` — Show current TQ configuration and compression stats

## Compression Profiles

### Standard (3-bit key / 2-bit value)
- **Compression:** 4.4x on full-attention layers
- **Key quality:** cos_sim = 1.000000 (near-lossless)
- **Value quality:** cos_sim = 0.940 (good for most workloads)
- **Use case:** Maximum VRAM savings, general inference

### Quality (3-bit key / 4-bit value)
- **Compression:** ~2.5x on full-attention layers
- **Key quality:** cos_sim = 1.000000
- **Value quality:** cos_sim = 0.997 (near-lossless)
- **Use case:** Quality-sensitive workloads, long-context reasoning

### Ultra (4-bit key / 4-bit value)
- **Compression:** ~2x on full-attention layers
- **Key quality:** cos_sim = 1.000000
- **Value quality:** cos_sim = 0.997
- **Use case:** Maximum quality preservation

## Benchmark Results (Verified)

### RTX 5090 (32GB) — Qwen3.5-27B-AWQ

| Metric | Baseline | TurboQuant | Gain |
|---|---|---|---|
| Prefill tok/s (30k ctx) | 1,804 | 1,907 | +5.7% |
| Decode tok/s (30k ctx) | 1.264 | 1.303 | +3.1% |
| KV cache freed | -- | 30.0 GB | -- |
| Max token capacity | 457,072 | 914,144 | **2.0x** |

### 8x RTX 3090 — Qwen3.5-35B-A3B MoE (TP=8)

| Context | Baseline KV/GPU | TQ KV/GPU | Savings |
|---|---|---|---|
| 32k | 191.5 MB | 132.3 MB | 30.9% |
| 64k | 374.3 MB | 258.5 MB | 30.9% |
| 131k | 755.7 MB | 521.9 MB | 30.9% |

MoE savings limited to 30.9% because only full-attention layers (10/40) are compressible. Pure dense transformers get **77% savings (4.4x)**.

## Architecture

```
turboquant/
  codebook.py          # Lloyd-Max optimal scalar quantizer
  codebooks/           # Pre-generated codebook files (d=128/256, bits 2-4)
  rotation.py          # Random orthogonal rotation + QJL projection
  quantizer.py         # TurboQuantMSE + TurboQuantProd (Algorithms 1 & 2)
  kv_cache.py          # KV cache manager with value bit-packing
  capture.py           # Modular KV capture hooks for attention layers
  store.py             # Compressed KV store (quantize + append + flat cache)
  score.py             # Attention scoring from compressed keys
  triton_kernels.py    # 3 fused Triton kernels for decode attention
  integration/vllm.py  # vLLM adapter (monkey-patch, hybrid decode)
```

## vLLM Integration

TurboQuant integrates with vLLM via monkey-patching:

```python
from turboquant.integration.vllm import patch_vllm_engine

# Patch a running vLLM engine
engine = LLMEngine.from_engine_args(args)
patch_vllm_engine(engine, key_bits=3, value_bits=2)

# KV cache is now compressed automatically
# No changes to API — prompts and completions work as before
```

## When To Use TurboQuant

| Scenario | Recommendation |
|---|---|
| Running out of VRAM on long contexts | **Yes** — 2x context capacity |
| Multiple concurrent users on one GPU | **Yes** — freed VRAM = more concurrent requests |
| Quality-critical (medical, legal, code) | Use 4-bit values (cos_sim=0.997) |
| Short contexts (<4k tokens) | **No** — overhead not worth it, KV is small |
| Linear-attention / Mamba models | **Limited** — only compresses full-attention layers |
| CPU inference | **No** — requires CUDA for Triton kernels |

## Known Limitations

- **Prefill uses paged cache** — TQ frees after prefill, not during. True zero-allocation needs deeper vLLM integration.
- **Only full-attention layers** — Linear-attention/Mamba layers are not compressed.
- **Value quantization bottleneck** — 2-bit values cause cos_sim=0.94. Use 4-bit for quality.
- **Hybrid decode dequantizes all history** — compressed tokens expand to float32 during compute.
- **Requires CUDA 12.8+** and PyTorch 2.10+ for Triton kernel support.

## Hardware Requirements

| Setup | Minimum | Recommended |
|---|---|---|
| GPU | RTX 3090 (24GB) | RTX 5090 (32GB) |
| CUDA | 12.8+ | 12.8+ |
| PyTorch | 2.10+ | 2.10+ |
| vLLM | 0.18.0+ | 0.18.0+ |
| Python | 3.12+ | 3.12+ |

## Paper Validation

9 tests validating ICLR 2026 paper theorems:

| Claim | Verdict |
|---|---|
| MSE distortion bounds (Thm 1) | PASS |
| Codebook MSE matches Table 1 | PASS |
| Unbiasedness (Thm 2) | PASS — relative bias < 0.1% |
| Distortion 1/4^b scaling (Thm 3) | PASS |
| Recall@8 (3-bit, N=4096) | 0.55 (threshold: >= 0.40) |
| Rank correlation (N=2048) | PASS — Spearman rho > 0.85 |
| Needle retrieval | PASS at all context lengths |
| Compression ratio | 4.41x at head_dim=256 |

## Honest Assessment

- "5.1x compression" claim is **misleading** — doesn't count Pi/S matrices. Honest: ~4.6x at 4k, ~5x at 32k+
- "Needle-in-haystack passes" is **true but trivial** — real queries are harder than query=key
- "30k TQ is faster" is **within noise** — N=1 run, TQ total wall time actually slightly slower
- The real value is **VRAM savings and context extension**, not speed