Results for “expert-parallel”
7 skills
orchestra-research moe-training
Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace, covering architectures, routing, load balancing, and expert parallelism.
10.4k · bundle
nvidia nemo-mbridge-perf-moe-comm-overlap
Optimizes MoE expert-parallel communication overlap in Megatron Bridge, covering dispatch/combine overlap, flex dispatcher backends, and expert wgrad scheduling.
2.2k · bundle
orchestra-research training-llms-megatron
Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies for maximum GPU efficiency.
10.4k · bundle
More results
nvidia nemo-mbridge-perf-moe-long-context
Provides guidance for training Mixture-of-Experts models with long context windows, covering context parallelism sizing, selective recomputation, dispatcher choices, and practical patterns from recent experiments.
2.2k · bundle
orchestra-research deepspeed
Provides expert guidance for distributed training with DeepSpeed, covering ZeRO optimization stages, pipeline parallelism, FP16/BF16/FP8, 1-bit Adam, and sparse attention.
10.4k · bundle
nvidia nemo-mbridge-perf-moe-vlm-training
Provides practical guidance for training Mixture-of-Experts Vision-Language Models in Megatron Bridge, comparing FSDP and 3D-parallel approaches with lessons from recent multimodal experiments.
2.2k · bundle
ichichuang pytorch-fsdp
Provides expert guidance on PyTorch Fully Sharded Data Parallel (FSDP) training, covering parameter sharding, mixed precision, CPU offloading, and FSDP2.
0 · bundle