Results for “distributed-training”

Launch distributed Megatron-LM training jobs on a SLURM cluster with a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules, container conventions, monitoring, and per-rank failure diagnosis.

Pretrains large language models at scale using PyTorch-native torchtitan with 4D parallelism, Float8, and distributed checkpointing.

Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision/offload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.

Train large language models (7B-70B+) with RLHF using PPO, GRPO, DPO, and other algorithms, accelerated by Ray and vLLM for distributed multi-GPU setups.

Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies for maximum GPU efficiency.

Scales machine learning training from single GPU to multi-node clusters with minimal code changes. Supports PyTorch, TensorFlow, and HuggingFace with built-in hyperparameter tuning, fault tolerance, and elastic scaling.

Provides expert guidance on PyTorch Fully Sharded Data Parallel (FSDP) training, covering parameter sharding, mixed precision, CPU offloading, and FSDP2.

Train reinforcement learning models using torchforge, Meta's PyTorch-native RL library for scalable, algorithm-focused experimentation with GRPO, DAPO, and custom loss functions.

Provides battle-tested PyTorch training recipes for LLMs, vision, diffusion, and biomedical domains, covering training loops, optimizer selection, LR scheduling, mixed precision, and debugging.

Train or fine-tune sentence-transformers models for retrieval, similarity, clustering, classification, and reranking, with support for bi-encoders, cross-encoders, and sparse encoders.

Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning) with support for SFT, DPO, GRPO, KTO, RLOO, and reward model training via CLI commands.

Finetunes the NV-Generate-CTMR MR-brain diffusion UNet from user-supplied NIfTI training volumes using a wrapper that stages configs and delegates to upstream scripts.

Fine-tune and align language models using reinforcement learning with TRL, including SFT, DPO, PPO, GRPO, and reward model training.

Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace, covering architectures, routing, load balancing, and expert parallelism.

Scales AI and Python applications across clusters with distributed computing primitives for ML workloads.

Convert single-node PyTorch distributed scripts into multi-node Slurm sbatch jobs and debug common multi-node failures, covering srun-native and torch.distributed approaches, container setup, NCCL timeouts, and interactive allocation.

Run ML training and batch jobs across multiple clouds with automatic cost optimization, spot instance recovery, and unified orchestration.

Manage and use Lambda Labs GPU cloud instances for ML training and inference with SSH access, persistent filesystems, and multi-node clusters.