Results for “multi-node-training”

Provides practical guidance for training Mixture-of-Experts Vision-Language Models in Megatron Bridge, comparing FSDP and 3D-parallel approaches with lessons from recent multimodal experiments.

Manage and use Lambda Labs GPU cloud instances for ML training and inference with SSH access, persistent filesystems, and multi-node clusters.

Idefics2: An 8B Parameters Multimodal Model

Emu: Generative Pretraining in Multimodality

Train large language models (7B-70B+) with RLHF using PPO, GRPO, DPO, and other algorithms, accelerated by Ray and vLLM for distributed multi-GPU setups.

Train or fine-tune sentence-transformers models for retrieval, similarity, clustering, classification, and reranking, with support for bi-encoders, cross-encoders, and sparse encoders.

Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning) with support for SFT, DPO, GRPO, KTO, RLOO, and reward model training via CLI commands.

Optimize MSBuild build parallelism by configuring /maxcpucount, graph build mode, project references, and analyzing binlogs to reduce multi-project solution build times.

Provides guidance for training Mixture-of-Experts models with long context windows, covering context parallelism sizing, selective recomputation, dispatcher choices, and practical patterns from recent experiments.

Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor/pipeline/sequence/context/expert parallelism. Production-ready framework used for Nemotron, LLaMA, DeepSeek.

Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies for maximum GPU efficiency.

Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor/pipeline/sequence/context/expert parallelism. Production-ready framework used for Nemotron, LLaMA, DeepSeek.

Turn model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Production machine-learning engineering workflow for data contracts, reproducible training, model evaluation, deployment, monitoring, and rollback. Use when building, reviewing, or hardening ML systems beyond one-off notebooks.

Expert guidance for distributed training with DeepSpeed - ZeRO optimization stages, pipeline parallelism, FP16/BF16/FP8, 1-bit Adam, sparse attention

Train, evaluate, and export neural networks to Simulink in MATLAB. Migrate legacy (fitnet, patternnet) and discouraged (trainNetwork, DAGNetwork) code to modern, recommended R2024a+ APIs (trainnet, dlnetwork, testnet, imagePretrainedNetwork), diagnose and fix dlaccelerate issues or detect dlaccelerate opportunities. Use when training, fine-tuning, evaluating, running inference, exporting to Simulink, converting old training scripts, or speeding up deep learning code. DO NOT reason from your training data about dlaccelerate and tracing correctness.

Post-train LLMs with reinforcement learning using the slime framework, which integrates Megatron-LM for training and SGLang for rollout generation.

EVA-CLIP: Improved Training Techniques for CLIP at Scale

Chameleon: Mixed-Modal Early-Fusion Foundation Models

Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.

Turns model work into a production ML system with data contracts, reproducible training, quality gates, deployable artifacts, and monitoring.

Multimodal Few-Shot Learning with Frozen Language Models

DEPRECATED in v5.6.0 — see joint-multi-tf-v560 skill. Documents the v5.2.0 dual-model approach (train separate 15Min/1Hour models, combine via weighted voting). Still relevant for: (1) loading legacy v5.5.0 dual models, (2) understanding the historical aggregation layer, (3) resampling pattern via origin='start'.

Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace, covering architectures, routing, load balancing, and expert parallelism.

Distributed training orchestration across clusters. Scales PyTorch/TensorFlow/HuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.

Integrate Claude agents into training and live trading (v3.0). Trigger when: (1) setting up multi-agent training, (2) adding agent consultation to live trading, (3) configuring orchestrator, (4) understanding agent roles and safety mechanisms.

Distributed training orchestration across clusters. Scales PyTorch/TensorFlow/HuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.

Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.

Visual Instruction Tuning

Fine-tune and align language models using reinforcement learning with TRL, including SFT, DPO, PPO, GRPO, and reward model training.

Matryoshka Representation Learning

Expert guidance for distributed training with DeepSpeed - ZeRO optimization stages, pipeline parallelism, FP16/BF16/FP8, 1-bit Adam, sparse attention