Simple Gpt Normalization Strategy

Place RMSNorm immediately after every linear layer to stabilize activation scales at O(sqrt(d)) and reduce Hessian spectral norm. Enables 3-10x larger learning rates and faster convergence without architectural changes; improves loss by 0.08 on 7B models.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit e1f091503e

Frequently asked questions

npx get-skill add adu2021/simple-gpt-normalization-strategy