Sparselora Contextual Sparsity Finetuning

Accelerates LoRA fine-tuning 2.2× computationally and 1.6× wall-clock by leveraging contextual sparsity to compute gradients only for important weight channels. Uses training-free SVD sparsity estimation without full computation. Apply for efficient fine-tuning on memory-constrained GPUs or large-scale training scenarios.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit fa90af29d4

Frequently asked questions

npx get-skill add adu2021/sparselora-contextual-sparsity-finetuning