Cached System Instructions

Cached System Instructions (KV-Cache Prefix Architecture)

Overview

In Large Language Models, processing input tokens requires running dense matrix multiplications across the entire input sequence on every turn. In a 50-turn agent session with a 4,000-token system prompt, the provider re-processes the exact same 4,000 tokens 50 separate times ($200,000\text{ redundant token computations}$).

Prompt Caching (Anthropic Prompt Caching, OpenAI Automatic Prefix Caching, DeepSeek Context Caching) saves the transformer's Key-Value (KV) Activation States in GPU memory. Re-sending an identical prefix reuses the precomputed KV cache, cutting Time-to-First-Token (TTFT) latency by 80% and applying an automatic 90% cost discount.

The Hierarchical System Caching Protocol organizes instructions to guarantee uninterrupted prefix matching.

Uncached Re-computation vs. KV-Cache Hit

┌─────────────────────────────────────────────────────────────┐
│                 KV-Cache Mechanics Comparison               │
│                                                             │
│  Uncached Execution (Every Turn):                           │
│  • Re-computes Attention Matrices for 4,000 System Tokens   │
│  • TTFT Latency: 2.8 seconds                                │
│  • Full Input Rate: $3.00 / M tokens                        │
│                                                             │
│  KV-Cache Hit (Unbroken Static Prefix):                     │
│  • Directly loads KV-tensors from GPU VRAM                  │
│  • TTFT Latency: 0.35 seconds (8x Faster!)                  │
│  • Cached Input Rate: $0.30 / M tokens (90% Cost Discount)  │
└─────────────────────────────────────────────────────────────┘

The 4-Layer Caching Hierarchy

To prevent accidental cache invalidation, place instructions in order of decreasing stability:

┌───────────────────────────────────────────────────────────────────────────┐
│ LAYER 1: Base Agent Identity & Tool Schemas (100% Immutable) ──► CACHED   │
│ LAYER 2: Repository Architecture Blueprint (Static for Session)──► CACHED │
│ LAYER 3: MCP Tool Definitions & Skill Instructions         ──► CACHED     │
│ ───────────────────────────────────────────────────────────────────────── │
│ LAYER 4: Dynamic User Requests & Shell Outputs (Dynamic Tail)──► UNCACHED │
└───────────────────────────────────────────────────────────────────────────┘

Production Anthropic SDK Caching Implementation

To activate prompt caching in Anthropic Claude 3.5 Sonnet / Haiku, place cache_control breakpoints at the end of large static blocks (minimum threshold: 1,024 tokens on Sonnet, 2,048 tokens on Haiku):

import anthropic

client = anthropic.Anthropic()

SYSTEM_PROMPT = """
You are Antigravity, an elite autonomous software engineering agent.
... [3,500 tokens of static rules, AST guidelines, and tool schemas] ...
"""

def query_agent_with_caching(user_query: str, history: list) -> str:
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=2048,
        system=[
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                # Set cache breakpoint on the static system prompt
                "cache_control": {"type": "ephemeral"}
            }
        ],
        messages=[
            *history,
            {"role": "user", "content": user_query}
        ]
    )
    
    # Inspect cache performance
    usage = response.usage
    print(f"Cache Created: {getattr(usage, 'cache_creation_input_tokens', 0)}")
    print(f"Cache Read (90% discount): {getattr(usage, 'cache_read_input_tokens', 0)}")
    return response.content[0].text

The 3 Golden Rules to Prevent Cache Busting

1. Never Inject Timestamps into the System Header: Dynamic strings like "Current Time: 2026-08-22T17:52:00" change every second and bust the entire prefix cache. Pass current time in the User turn.
2. Deterministic Tool Serialization: Ensure tool schemas are serialized with sorted JSON keys (json.dumps(obj, sort_keys=True)) to maintain bit-for-bit string equality.
3. Keep the Head Static: Never insert dynamic project names at line 1. Put project names at the bottom of the system block or in the first user message.

Benchmark Metrics

Evaluation across a 30-turn autonomous feature build (4,000-token system prompt):

Agent Operational Directive

MANDATORY: All system prompts exceeding 1,000 tokens must be structured as immutable, deterministic headers with prompt-cache control breakpoints enabled. Never inject dynamic runtime variables into the top-level system prefix.