Adaptive Speculative Decoding

Accelerate LLM inference 1.5-2x using a universal draft model that adapts to user data in real-time, handling different target models and tokenizers via online n-gram cache and hybrid distillation.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 22aeaf33bd

Frequently asked questions

npx get-skill add adu2021/adaptive-speculative-decoding