Sonicmoe Training Optimization

Optimize MoE training through memory-efficient backward pass, IO-aware kernel design overlapping memory operations, and token rounding routing. Avoid caching large-scale activations, fuse operations with GEMM, implement ping-pong scheduling. Achieve 1.86× compute throughput on Hopper GPUs and 16% higher TFLOPS in sparse configurations.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 2c09601b18

Frequently asked questions

npx get-skill add adu2021/sonicmoe-training-optimization