Learnable Multipliers Lm Scaling

Adapt language model weight matrix scales automatically during training by adding learnable scalar and vector multipliers to each layer. Breaks the noise-weight-decay equilibrium that constrains scales based on hyperparameters rather than data, enabling data-driven scaling adaptation without inference cost or extensive tuning overhead.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 35fadb32d1

Frequently asked questions

npx get-skill add adu2021/learnable-multipliers-lm-scaling