Flowprefill Scheduling Preemption

Improve LLM serving under mixed workloads by decoupling execution granularity from scheduling frequency. Operator-level preemption allows fine-grained interruption at natural boundaries (attention, feed-forward layers) without efficiency loss. Event-driven scheduling triggers decisions only on request arrival/completion. Eliminates head-of-line blocking where long requests starve short time-sensitive ones. Achieves 5.6× higher goodput vs. baselines in production workloads.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit e8fb777a9f

Frequently asked questions

npx get-skill add adu2021/flowprefill-scheduling-preemption