Flash Sampling Efficient Decoding

Fuse categorical sampling directly into LM-head matrix multiplication to eliminate logits materialization. Use Gumbel noise during computation and hierarchical reduction to achieve 19% token-level latency reduction.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 8ac658945f

Frequently asked questions

npx get-skill add adu2021/flash-sampling-efficient-decoding