Muon Optimizer Tail Memory Learning

Improve LLM training efficiency by selectively applying Muon optimizer to Value-Output attention weights and FFN layers, which function as associative memories. Use when training data exhibits heavy-tailed distributions requiring robust rare-fact learning.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit ba28d50ccb

Frequently asked questions

npx get-skill add adu2021/muon-optimizer-tail-memory-learning