adu2021 Overview
Reformulates long-context as continual learning problem solved at test time.
Core Technique
Meta-Learning for Test Time:
# Inner loop: compress context into weights
for token in context:
gradient = compute_gradient_on_token(model, token)
model.weights += gradient # Compress context
# Outer loop: optimize initialization
# Treat inner loop as differentiable step
Performance
- Full-attention quality across context lengths
- 2.7× faster than attention at 128K
- Constant-time inference
References
- Test-time meta-learning
- Weight compression of context
- End-to-end optimization