Progressive Residual Warmup

Improves LLM convergence and downstream task performance by introducing time-dependent scaling to residual connections, enabling shallow layers to learn first before deeper layers activate. Apply during model pretraining to achieve 0.4-4.86 perplexity reduction.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 0394329dcd

Frequently asked questions

npx get-skill add adu2021/progressive-residual-warmup