Mixture Of Depths Attention

Allow attention heads to reference features from multiple depths by accessing both current-layer and depth key-value pairs. Prevent signal degradation in deep models while maintaining computational efficiency.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit b5fd99150d

Frequently asked questions

npx get-skill add adu2021/mixture-of-depths-attention