adu2021 Muon Optimizer: Selective Parameter Optimization for Associative Memory
This research explains why Muon optimizer outperforms Adam through mechanistic analysis. Muon excels at learning transformer components functioning as associative memories—specifically Value-Output attention and FFNs—particularly when training data has heavy-tailed distributions where rare facts appear infrequently.
Core Architecture
- Selective application: Apply Muon to VO+FFN, Adam to other parameters
- Associative memory insight: VO weights and FFNs implement content-addressable memory
- Heavy-tailed robustness: Muon handles rare but important facts better than Adam
- Computational efficiency: Near-full gains with only 40% of parameters using Muon
Implementation Steps
Apply selective optimizer assignment to transformer layers:
# Create mixed optimizer configuration targeting specific components
from torch.optim import Adam
from muon_optim import Muon
def create_selective_optimizer(model, learning_rate=1e-3):
"""Assign Muon to VO+FFN, Adam to remaining parameters"""
vo_ffn_params = []
other_params = []
for name, param in model.named_parameters():
if any(x in name for x in ["value_out", "ffn"]):
vo_ffn_params.append(param)
else:
other_params.append(param)
optimizer = torch.optim.Adam([
{
"params": vo_ffn_params,
"optimizer": Muon,
"lr": learning_rate,
"momentum": 0.95
},
{
"params": other_params,
"optimizer": Adam,
"lr": learning_rate,
"betas": (0.9, 0.999)
}
])
return optimizer
Execute training with mixed optimization:
# Standard training loop with selective optimizer
optimizer = create_selective_optimizer(model)
for step, (x, y) in enumerate(dataloader):
logits = model(x)
loss = compute_loss(logits, y)
# Standard backprop; optimizer handles mixed updates
loss.backward()
optimizer.step()
optimizer.zero_grad()
Practical Guidance
When to use selective Muon:
- Training language models with heavy-tailed knowledge distributions
- Models requiring robust handling of rare facts or long-tail phenomena
- Scenarios where compute budget allows selective optimization
- Tasks combining frequent patterns with rare exceptions
When NOT to use:
- Uniform data distributions (Adam sufficient)
- Domains where all parameters equally important
- When full-model Muon training already optimal
- Real-time deployment with strict latency constraints
Hyperparameters:
- Momentum (0.95): Standard for Muon; increase to 0.97 for slower convergence, decrease to 0.9 for stability
- Learning rate: Keep Muon and Adam learning rates equal initially; adjust Muon upward by 1.5x if underfitting
- Parameter allocation: 40% Muon (VO+FFN) optimal across scales; test 30-50% range for different architectures
Component selection:
- VO (Value-Output) weights: Always include; these implement fact retrieval
- FFN layers: Include entirely or FFN-up only depending on model design
- Attention scores: Keep with Adam; norm-based updates less suitable
- Embeddings: Keep with Adam; Muon offers limited benefit
Architecture Notes
The VO+FFN pair together form transformer associative memory:
- VO weights map from attention-computed queries to output space
- FFN layers implement non-linear fact association
- Heavy-tailed data (rare facts) benefits from Muon's orthonormality-preserving updates
Performance Improvements
- Near-full gains (~95% of single-Muon performance) with selective optimization
- Faster training due to reduced Muon computation on fewer parameters
- Robust scaling from 160M to 700M parameter models
References
Builds on mechanistic interpretability of transformer components and optimizer theory.