Amey-Thakur Model parallelism
When a model is too large for one GPU, you cut it apart, and every cut buys communication. Tensor, pipeline, and expert parallelism split along different axes, each with its own network cost and its own way of leaving GPUs idle. Choose badly and the cards wait on an all-reduce or sit in a pipeline bubble, so the decision is really about matching the split to your interconnect and batch shape.
Method
- Compute the memory shortfall before choosing anything. Weight bytes are roughly parameters times dtype bytes: a 70B model in FP16 wants about 140 GB and will not fit one 80 GB H100. For training, add optimizer state and activations. The gap sets the minimum GPU count ahead of any strategy.
- Keep tensor parallelism inside one fast domain. Tensor parallelism (Megatron style) splits each matrix multiply across GPUs and all-reduces twice per layer, so it demands high bandwidth. Hold the tensor-parallel group within a single NVLink node, usually 8 GPUs, and never stretch it over InfiniBand.
- Use pipeline parallelism to cross node boundaries. Pipeline parallelism assigns whole layer ranges to different GPUs and passes activations point to point, cheap enough for inter-node links. Its cost is the pipeline bubble: the idle time at the head and tail of every batch.
- Cut the bubble with more microbatches. The bubble fraction is about (stages - 1) / microbatches. Four stages with four microbatches waste near 43 percent; sixteen microbatches bring that under 20 percent. Raise the count until the bubble is small or activation memory runs out.
- Reach for expert parallelism only in a mixture-of-experts model. MoE routes each token to a few experts, so you scatter experts across GPUs and pay an all-to-all to shuffle tokens. It grows parameter count without growing per-token compute, but the all-to-all is bursty and punishes routing imbalance.
- Factor the strategies as a grid, not a heap. Real runs combine them: tensor parallel within a node, pipeline parallel across nodes, data parallel over the whole. Write the factorization out, for example TP=8, PP=4, DP=2 for 64 GPUs, so the product equals your total device count exactly.
- Measure the interconnect before you trust the plan. Run an NCCL bandwidth test and confirm NVLink and InfiniBand hit their expected GB/s. A tensor parallel group that accidentally crosses a PCIe hop instead of NVLink can halve throughput while looking correctly configured on paper.
Litmus tests
- Does the tensor-parallel group stay inside one NVLink node?
- Is the microbatch count high enough to keep the pipeline bubble under 20 percent?
- Does TP times PP times DP equal your total GPU count exactly?
- Did an NCCL bandwidth test confirm each group runs over the link you assumed?
Boundaries
This is about splitting one model across devices. Replicating a model that does fit and overlapping gradient communication is distributed-training-scaling; sizing and running a single serving replica is vllm-serving. The partition that is optimal depends on your model shape and cluster topology, which is a matter of measurement, not doctrine.