Mu Parametrization Moe

Apply μ-parametrization to Mixture-of-Experts architectures to enable reliable hyperparameter transfer across model sizes, eliminating costly retuning when scaling to trillion-parameter systems.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 9e07a9622f

Frequently asked questions

npx get-skill add adu2021/mu-parametrization-moe