Model Parallelism

Split a model that will not fit on one GPU across devices with tensor, pipeline, and expert parallelism, choosing each split by its communication cost. Use when weights or activations exceed a single card and you must shard without stalling on the interconnect.

Amey-Thakur Updated 14 repo stars

File contents

Amey-Thakur/AI-SKILLS commit b43469eab4

Frequently asked questions

npx get-skill add amey-thakur/model-parallelism