Hybrid Linear Attention

Design hybrid architectures mixing linear and full attention at optimal ratios. Superior standalone linear models don't necessarily excel in hybrids; recall improves significantly with more full attention layers at ratios below 3:1, enabling efficient long-sequence processing.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 117ba623f1

Frequently asked questions

npx get-skill add adu2021/hybrid-linear-attention