Let Late Early Training

Accelerate LLM pretraining by leveraging small pretrained models as teachers, enabling large models to explicitly learn late-layer knowledge in early layers via alignment loss, achieving 1.6x convergence speedup and 5% downstream improvement even with 10x smaller teachers.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 33558d42a9

Frequently asked questions

npx get-skill add adu2021/let-late-early-training