Minicpm Sala

Combine sparse attention (25% of layers) and linear attention (75% of layers) via strategic layer placement to handle 1M-token contexts with 75% training cost reduction. Hybrid positional encoding preserves long-range information while maintaining position awareness.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit f0211b1d54

Frequently asked questions

npx get-skill add adu2021/minicpm-sala