Torch Pipeline Parallelism

This skill provides guidance for implementing PyTorch pipeline parallelism for distributed training of large language models. It should be used when implementing pipeline parallel training loops, partitioning transformer models across GPUs, or working with AFAB (All-Forward-All-Backward) scheduling patterns. The skill covers model partitioning, inter-rank communication, gradient flow management, and common pitfalls in distributed training implementations.

lazyfroglol Updated 129 repo stars

File contents

lazyfroglol/harness_engineering commit e2d43a7e5e

Frequently asked questions

npx get-skill add lazyfroglol/torch-pipeline-parallelism