Mirror Speculative Decoding

Run draft and target models in parallel across GPU/NPU using bidirectional speculation: draft predicts forward continuations while target speculates correction paths simultaneously. Achieve 2.8x-5.8x wall-time speedup on 14B-66B models.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 1a2fd2e42b

Frequently asked questions

npx get-skill add adu2021/mirror-speculative-decoding