Sglang Radix Attention

/sglang-radix-attention

When to Use

• RAG workloads: same system prompt + same retrieval preamble + varying question
• Agentic workloads: same tool schemas + system prompt repeated across thousands of requests
• Chat with long, stable system prompts (>1K tokens)
• Voice/vision with repeated preambles (audio feature header, image encoder prefix)

Do NOT use for

• Open-ended chat with unique prompts per request (no shared prefix = no cache benefit)
• Single-shot code completion where every prompt is distinct
• Short prompts (<200 tokens) where cache lookup overhead exceeds savings

How RadixAttention works

The radix tree stores token sequences as paths from root to leaf.
Each node owns KV blocks for the token range on its path.
A new request WALKS the tree: matching nodes → KV blocks reused → no prefill.

Example tree:
  root
   └─ "You are a helpful assistant..." (2,000 tokens = 124 KV blocks)
       ├─ "Context: <doc A>..." (500 tokens, 31 blocks)
       │   ├─ "Question: Alice"  (80 tokens, 5 blocks — NEW: only 5 blocks prefilled)
       │   └─ "Question: Bob"    (95 tokens, 6 blocks)
       └─ "Context: <doc B>..." (520 tokens, 33 blocks)

Without tree: request for doc-A + Alice prefills 2,625 tokens (160 blocks).
With tree:    prefills only 80 tokens (5 blocks). ~32x prefill savings.

The ordering invariant (engineer's key lever)

The 6.4x improvement REQUIRES consistent prompt template ordering.
A shared prefix to a human ≠ shared prefix to the radix tree.

[system, tools, context, history, question]   ← correct (immutable first)
[system, context, tools, history, question]   ← different order → tree miss

Rule: put everything immutable (system, tools, schemas) FIRST.
      put retrieval context next.
      put user question LAST.
      NEVER interleave dynamic content into the stable prefix.

Real case: one deployment went from 7% → 74% cache hit rate
           by moving dynamic content out of the cacheable prefix.

Cache-aware scheduling

Problem with FCFS (first-come, first-served):
  Request A shares 2,000-token prefix.
  Request B arrives first, shares only 50 tokens.
  FCFS serves B → 2,000-token branch evicted before A is served.

Cache-aware scheduling policies:
  1. Depth-first dispatch    — prefer requests rooted at the same branch as running set
  2. LRU at branch level     — evict whole branches (not individual blocks), shortest-used first

Benchmark numbers to remember

Benchmark                                       SGLang vs vLLM
──────────────────────────────────────────────  ──────────────────
Llama 3.1 8B, H100, ShareGPT 1K prompts         ~16,200 vs ~12,500 tok/s  (+29%)
Prefix-heavy RAG (same doc, varying question)    up to 6.4x
Voice cloning workloads                          86.4% prefix cache hit rate
Production hit rates across customers            50–99% (depends on prompt discipline)
Deployment scale (2026)                          400,000+ GPUs

Use sGLang over vLLM when: prefix-heavy workload + prompt ordering is controlled.
Use vLLM when: unique-prompt workloads or you need PagedAttention's continuous batching.

sGLang deployment (Docker)

# Start sGLang server (OpenAI-compatible API)
docker run --gpus all \
  -p 30000:30000 \
  lmsysorg/sglang:latest \
  python -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --port 30000 \
    --enable-prefix-caching \
    --cache-radix-tree \
    --mem-fraction-static 0.88

# OpenAI-compatible: point any client at http://localhost:30000

Prompt template discipline (Python)

from dataclasses import dataclass

@dataclass
class PromptBuilder:
    """Enforces immutable-first ordering for maximum prefix cache hits."""
    system_prompt: str          # IMMUTABLE — goes first
    tool_schemas:  list[str]    # IMMUTABLE — goes second
    few_shot_examples: list[str] = ()  # IMMUTABLE if fixed

    def build(self, context: str, history: list[str], question: str) -> str:
        parts = [
            f"[SYSTEM]\n{self.system_prompt}",
            "[TOOLS]\n" + "\n".join(self.tool_schemas),
        ]
        if self.few_shot_examples:
            parts.append("[EXAMPLES]\n" + "\n".join(self.few_shot_examples))
        # Dynamic parts LAST — do not reorder
        if context:
            parts.append(f"[CONTEXT]\n{context}")
        if history:
            parts.append("[HISTORY]\n" + "\n".join(history))
        parts.append(f"[QUESTION]\n{question}")
        return "\n\n".join(parts)

    def validate_ordering(self) -> None:
        """Fail fast if dynamic content would corrupt the immutable prefix."""
        if "{" in self.system_prompt or "{" in " ".join(self.tool_schemas):
            raise ValueError("System prompt and tool schemas must not contain dynamic f-string placeholders")

Anti-Fake-Pass Checklist

❌ Dynamic content interleaved into system prompt → radix tree cannot find shared prefix → 6.4x → 1.0x
❌ Prompt template varies across requests → cache miss rate > 90% → sGLang offers no advantage over vLLM
❌ FCFS scheduling not replaced → hot branch evicted before high-prefix-share requests → cache churns
❌ All-in-one vLLM used for RAG workload → missing 29–6.4x throughput; switch to sGLang
❌ Short prompts (<200 tokens) → lookup overhead > savings; RadixAttention not worth it
❌ Assuming "cache hit" without measuring → instrument with SGLang's cache_hit_rate metric