Specontext Speculative Caching

Uses distilled language model (DLM) for KV cache token retrieval, reducing layer-wise retrieval parameters by 90% via head-level attention weights, with asynchronous prefetching and elastic CPU offloading. Deploy for ultra-long-context inference with memory constraints.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 32db7b529a

Frequently asked questions

npx get-skill add adu2021/specontext-speculative-caching