Torch Tensor Parallelism

This skill provides guidance for implementing tensor parallelism in PyTorch, specifically column-parallel and row-parallel linear layers. Use when implementing distributed neural network layers that split weights/activations across multiple ranks, working with torch.distributed for model parallelism, or implementing ColumnParallelLinear and RowParallelLinear classes.

lazyfroglol Updated 129 repo stars

File contents

lazyfroglol/harness_engineering commit e555efcc9f

Frequently asked questions

npx get-skill add lazyfroglol/torch-tensor-parallelism