Results for “llm-pretraining”

Pretrains large language models from scratch using PyTorch-native distributed training with 4D parallelism (FSDP2, TP, PP, CP) and Float8 support on H100 GPUs.

Provides battle-tested PyTorch training recipes for LLMs, vision, diffusion, and biomedical domains, covering training loops, optimizer selection, LR scheduling, mixed precision, and debugging.

Post-train LLMs with reinforcement learning using the slime framework, which integrates Megatron-LM for training and SGLang for rollout generation.

Guides LLM post-training with RL using slime, a Megatron+SGLang framework for training GLM, Qwen, DeepSeek, and Llama models with GRPO, async, and multi-turn workflows.

Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data, covering correlation testing, available recipes, and multi-GPU examples.

Validate and configure packed sequences and long-context training in Megatron-Bridge, distinguishing offline packed SFT for LLMs from in-batch packing for VLMs with correct context parallelism constraints.

Entrena modelos de machine learning con Scikit-learn, LightGBM y XGBoost, desde un baseline hasta un modelo productivo con validación robusta y explicabilidad.

Train, fine-tune, and deploy LLMs using LitGPT's clean implementations of 20+ architectures like Llama, Gemma, and Phi.

Guides machine learning development with experiment tracking, hyperparameter optimization, model registry, and MLOps pipeline integration.

Trains LLMs with SimPO, a reference-free preference optimization method that outperforms DPO, using configurable hyperparameters and workflows for various models and tasks.

Train large-scale MoE models with FP8/INT4 low-precision RL, speculative decoding, and train-inference alignment using the miles framework.

Organizes PyTorch code with a Trainer class, automatic distributed training (DDP/FSDP/DeepSpeed), callbacks, and minimal boilerplate. Scales from laptop to supercomputer with the same code.

Compress large language models using knowledge distillation from teacher to student models, covering temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.