Token Sparse Attention Long Context

Dynamically select important tokens at the attention head level, performing dense attention only on selected tokens and scattering results back. Achieves 3.23x attention speedup at 128K context with 1% accuracy loss through layer-wise representation stability analysis.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 7fb6aa5b3e

Frequently asked questions

npx get-skill add adu2021/token-sparse-attention-long-context