Causal Attention Lookahead Keys

Implement CASTLE, a causal attention mechanism that dynamically updates key representations as context expands. Reduces validation loss by 0.006-0.037 across model scales while maintaining O(L²d) training complexity and O(td) decoding speed. Deploy for improved language model perplexity without inference overhead.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit c22502ef46

Frequently asked questions

npx get-skill add adu2021/causal-attention-lookahead-keys