Growing Transformers Modular Expansion

Build language models layer-by-layer on frozen embeddings, training new Transformer blocks sequentially while keeping lower layers frozen. Achieves 5% improvement over monolithic baselines on MMLU while fitting 740M trainable parameters per stage on single GPUs, enabling resource-efficient incremental scaling to 2.3B parameters.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit fa9d2f8a33

Frequently asked questions

npx get-skill add adu2021/growing-transformers-modular-expansion