Kvcache Format

KV-cache page format

See agent_docs/cache-layer.md for the full spec. Cheat sheet:

On-GPU page

A page is a contiguous [n_layers, page_size_tokens, n_heads, head_dim] tensor for K, plus the same shape for V. Stored in vLLM's gpu_cache as a list of (K_page, V_page) pairs indexed by physical-page id.

Page-out

for phys_pid in occupied_pages:
    K, V = read_page(phys_pid)              # GPU → pinned host buffer
    digest = sha256(K.bytes() ‖ V.bytes())  # one digest per (K,V) pair
    if not blob_store.contains(digest):
        compressed = zstd_19(K.bytes() ‖ V.bytes())
        blob_store.put_with_digest(compressed, digest)
    page_table[phys_pid] = digest
emit_manifest({page_size_tokens, n_layers, n_heads, head_dim, dtype, pages: page_table})

Page-in

manifest = read_manifest(cid)
for phys_pid, digest in zip(allocate_pages(len(manifest.pages)), manifest.pages):
    compressed = blob_store.get(digest)
    K_bytes, V_bytes = split(zstd_decompress(compressed))
    write_page(phys_pid, K_bytes, V_bytes)  # pinned → GPU
patch_logical_seq_table(manifest.logical_seqs)

Logical sequences

vLLM tracks per-request logical-seq → physical-page mappings. We serialize that mapping into logical_seqs[] so restore can rebuild request-level prefix-sharing exactly.

Bit-exactness

Requires the engine started with batch-invariant kernels (vLLM --enforce-deterministic, SGLang --deterministic-mode). Otherwise restore will produce numerically equivalent but not bit-equal logits.