adu2021 Skill Summary
Rather than retraining from scratch, this method systematically combines five complementary adaptation strategies to convert full-attention LLMs to sliding window attention (SWA). The key insight is that "no single ingredient suffices"—specific synergistic combinations of these methods effectively recover original performance. The approach identifies configurations achieving substantial speedups while preserving accuracy, depending on whether efficiency or quality takes priority.
When To Use
- Converting existing full-attention models to efficient sliding window attention
- Scenarios requiring 30-100% inference speedups with minimal accuracy loss
- Long-context applications where full attention is computationally prohibitive
- Projects already deployed with full-attention models seeking efficiency gains
When NOT To Use
- Models requiring full global attention for certain reasoning patterns
- Tasks where 10% accuracy loss is unacceptable (some configurations may not preserve perfect accuracy)
- Scenarios where the 5 adaptation strategies contradict your specific architectural constraints
- Models designed for short sequences where attention is already efficient
Core Technique
The approach systematically combines five complementary strategies:
1. Full Attention Decode Apply SWA only during prefilling (context encoding), switch to full attention during generation when sequence lengths are short.
2. Keep First k Tokens Preserve attention to initial "sink" tokens while using SWA elsewhere, maintaining access to document-level context.
3. Interleaving FA/SWA Layers Alternate between full-attention and SWA layers throughout the model, balancing efficiency and global context awareness.
4. Chain-of-Thought Enable explicit reasoning during decoding to compensate for limited context windows, allowing the model to compensate for SWA context constraints.
5. Fine-tuning Lightweight SWA-aware supervised fine-tuning on long-context data to adapt the model to the new attention pattern.
Implementation Notes
Start with your full-attention model. Experiment with combinations of these five strategies:
- For maximum speed: use SWA prefill + SWA decode + interleaved layers
- For maximum quality: use full decode + keep-first-k + chain-of-thought + fine-tuning
- For balanced approach: use SWA prefill + full decode + interleaved layers + light fine-tuning
Recommended: Test recommended configurations on your specific task to find the right accuracy-efficiency trade-off.
References
- Original paper: SWAA (Dec 2025)
- Sliding window attention mechanisms
- Long-context fine-tuning strategies