Results for “megatron-core”

Enables Megatron Fully Sharded Data Parallel in Megatron-Bridge with configuration overrides, code anchors, pitfalls, and verification steps.

Launch distributed Megatron-LM training jobs on a SLURM cluster with a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules, container conventions, monitoring, and per-rank failure diagnosis.

Reduces peak GPU memory in Megatron Bridge training by applying expandable segments, parallelism resizing, activation recompute, and CPU offloading constraints.

Configure and validate CPU offloading for Megatron Bridge training, including activation offloading and optimizer state offloading with HybridDeviceOptimizer.

Validate and use CUDA graph capture in Megatron Bridge, including local full-iteration graphs and Transformer Engine scoped graphs for attention, MLP, and MoE modules.

Indexes Megatron Bridge recipes and recommends the best starting config based on model, GPU count, and training goal.

Plan, configure, and chain Nemotron model customization steps into single-step or multi-step pipelines for curation, translation, fine-tuning, RL alignment, benchmarking, checkpoint conversion, optimization, and evaluation.

Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace, covering architectures, routing, load balancing, and expert parallelism.

Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data, covering correlation testing, available recipes, and multi-GPU examples.

Validate and configure packed sequences and long-context training in Megatron-Bridge, distinguishing offline packed SFT for LLMs from in-batch packing for VLMs with correct context parallelism constraints.

Plan, debug, tune, evaluate, export, or deploy public Nemotron embedding and reranking retrieval recipes using the current checkout.

Configure fault tolerance, straggler detection, preemption, in-process restart, and re-run state machine for Megatron Bridge training jobs.

Train, evaluate, export, and run inference for Masked Auto-Encoder (MAE) models for self-supervised pretraining and fine-tuning of visual representations.

Optimizes MoE expert-parallel communication overlap in Megatron Bridge, covering dispatch/combine overlap, flex dispatcher backends, and expert wgrad scheduling.

Post-train LLMs with reinforcement learning using the slime framework, which integrates Megatron-LM for training and SGLang for rollout generation.

Guides LLM post-training with RL using slime, a Megatron+SGLang framework for training GLM, Qwen, DeepSeek, and Llama models with GRPO, async, and multi-turn workflows.