Noloco Low Communication Training

Scale distributed LLM training without all-reduce synchronization using dynamic pipeline routing and modified Nesterov momentum, achieving 4% faster convergence than DiLoCo with exponentially lower communication.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit a8796eebfc

Frequently asked questions

npx get-skill add adu2021/noloco-low-communication-training