Batch Speculative Decoding

Fixes batch speculative decoding ragged tensor problem where sequences in batches accept different token counts, desynchronizing state. EQSPEC guarantees output equivalence through proper synchronization. EXSPEC reduces overhead 40% via cross-batch scheduling. Enables efficient parallel decoding with 95% equivalence.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 91229a3e5c

Frequently asked questions

npx get-skill add adu2021/batch-speculative-decoding