Dualpath Storage Bandwidth Optimization

Optimize disaggregated prefill-decoding LLM serving for multi-turn (agentic) workloads by introducing dual-path KV-cache loading. Traditional approach loads all KV-cache to prefill engines, saturating their storage network. DualPath loads to decoding engines first, then transfers via compute network (lower contention). Adaptive routing selects path based on real-time queue depths. Achieves 1.87× offline throughput and 1.96× online serving improvement.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 3cebecd839

Frequently asked questions

npx get-skill add adu2021/dualpath-storage-bandwidth-optimization