Distributed LLM Pretraining Torchtitan

Pretrains large language models from scratch using PyTorch-native distributed training with 4D parallelism (FSDP2, TP, PP, CP) and Float8 support on H100 GPUs.

Orchestra Research Updated 14.5k repo stars
AI Research Skills Demo

File contents

Orchestra-Research/AI-Research-SKILLs commit 773a52944b

Frequently asked questions

npx get-skill add orchestra-research/distributed-llm-pretraining-torchtitan