LLM Inference Batching Scheduler

Guidance for implementing batching schedulers for LLM inference systems with compilation-based accelerators. This skill applies when optimizing request batching to minimize cost while meeting latency thresholds, particularly when dealing with shape compilation costs, padding overhead, and multi-bucket request distributions. Use this skill for tasks involving batch planning, shape selection, generation-length bucketing, and cost-model-driven optimization for neural network inference.

lazyfroglol Updated 129 repo stars

File contents

lazyfroglol/harness_engineering commit a64f9d9192

Frequently asked questions

npx get-skill add lazyfroglol/llm-inference-batching-scheduler