Log Linear Sparse Attention

Reduce self-attention complexity from O(N²) to O(N log N) through hierarchical token selection and enrichment. Perform hierarchical Top-K selection progressively adopting sparse Top-K at each level. Implement sparse index transpose algorithm avoiding dense mask construction. Achieve 28.27× faster inference and 6.09× faster training.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit f446d91274

Frequently asked questions

npx get-skill add adu2021/log-linear-sparse-attention