Results for “torch-distributed”

Train reinforcement learning models using torchforge, Meta's PyTorch-native RL library for scalable, algorithm-focused experimentation with GRPO, DAPO, and custom loss functions.

Add distributed training support to any PyTorch script with minimal code changes using a unified API for DDP, DeepSpeed, FSDP, and mixed precision.

Organizes PyTorch code with a Trainer class, automatic distributed training (DDP/FSDP/DeepSpeed), callbacks, and minimal boilerplate. Scales from laptop to supercomputer with the same code.

Adds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision/offload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.

Organize PyTorch code into LightningModules, configure Trainers for multi-GPU/TPU, implement data pipelines, callbacks, logging (W&B, TensorBoard, MLflow), and distributed training (DDP, FSDP, DeepSpeed) for scalable neural network training.

Provides expert guidance on PyTorch Fully Sharded Data Parallel (FSDP) training, covering parameter sharding, mixed precision, CPU offloading, and FSDP2.

Provides guidance on using PyTorch for deep learning, covering tensors, autograd, nn.Module, DataLoaders, and best practices.

Launch distributed Megatron-LM training jobs on a SLURM cluster with a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules, container conventions, monitoring, and per-rank failure diagnosis.

Train, evaluate, distill, quantize, export, and run inference for RT-DETR object detection models using NVIDIA TAO.

Convert single-node PyTorch distributed scripts into multi-node Slurm sbatch jobs and debug common multi-node failures, covering srun-native and torch.distributed approaches, container setup, NCCL timeouts, and interactive allocation.

Build and train graph neural networks for drug discovery, protein modeling, and molecular science using PyTorch-native tools.

Optimizes transformer attention with Flash Attention for 2-4x speedup and 10-20x memory reduction. Supports PyTorch native SDPA, flash-attn library, H100 FP8, and sliding window attention.

Scales machine learning training from single GPU to multi-node clusters with minimal code changes. Supports PyTorch, TensorFlow, and HuggingFace with built-in hyperparameter tuning, fault tolerance, and elastic scaling.