Distributed Training Scaling

Scale training from one GPU to many by moving through data parallel and sharded FSDP modes, overlapping communication with compute, and reading the efficiency curve to find the ceiling. Use when adding GPUs stops making training proportionally faster and you need to locate where the scaling leaks.

Amey-Thakur Updated 14 repo stars

File contents

Amey-Thakur/AI-SKILLS commit ebbfb2b03a

Frequently asked questions

npx get-skill add amey-thakur/distributed-training-scaling