Scaling Dora Factored Norms

Optimize adapter parameter efficiency at scale by decomposing row-wise norm computation into base/cross/BA components (15× memory reduction) and fusing kernel operations. Achieves 1.5–2.0× inference speedup with 77 GB peak VRAM reduction across 8–32B vision-language models; applies when training adapter-based models with strict memory budgets across hundreds of modules.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 7da51a8244

Frequently asked questions

npx get-skill add adu2021/scaling-dora-factored-norms