Results for “mocha”

Provides guidance on using PyTorch for deep learning, covering tensors, autograd, nn.Module, DataLoaders, and best practices.

Provides guidance for training Mixture-of-Experts models with long context windows, covering context parallelism sizing, selective recomputation, dispatcher choices, and practical patterns from recent experiments.

Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace, covering architectures, routing, load balancing, and expert parallelism.

Implements Manifold-Constrained Hyper-Connections (mHC) using Doubly Stochastic Matrices to improve deep learning stability.