Results for “expert-parallelism”
4 skills
orchestra-research Moe Training
Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace, covering architectures, routing, load balancing, and expert parallelism.
10.4k · bundle
orchestra-research Training Llms Megatron
Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies for maximum GPU efficiency.
10.4k · bundle
More results
nvidia Nemo Mbridge Perf Moe Long Context
Provides guidance for training Mixture-of-Experts models with long context windows, covering context parallelism sizing, selective recomputation, dispatcher choices, and practical patterns from recent experiments.
2.2k · bundle
orchestra-research Deepspeed
Provides expert guidance for distributed training with DeepSpeed, covering ZeRO optimization stages, pipeline parallelism, FP16/BF16/FP8, 1-bit Adam, and sparse attention.
10.4k · bundle