Serving Frameworks

Expert knowledge of modern LLM/ML inference serving — vLLM (PagedAttention, continuous batching), SGLang (RadixAttention), NVIDIA Dynamo (disaggregated prefill/decode, KV-aware routing), Triton Inference Server, TensorRT-LLM, Ray Serve, KServe, JetStream (JAX/XLA on TPU), NVIDIA NIM (prepackaged inference microservices), and TGI. Use when deploying, tuning, or choosing an inference engine; reasoning about prefill vs decode, KV cache / PagedAttention / prefix caching, continuous/in-flight batching, TTFT/ITL/TPOT/goodput, tensor/pipeline/expert parallelism, multi-host serving, disaggregated serving, speculative/chunked-prefill decoding, fp8/AWQ/GPTQ quantization, structured/guided output, or serving autoscaling and KV-cache-aware routing on Kubernetes/GKE. Covers engine differentiators, a decision matrix, and K8s deployment shapes.

sanjeevrg89 Updated 0 repo stars

File contents

sanjeevrg89/arete commit 724d56aacf

Frequently asked questions

npx get-skill add sanjeevrg89/serving-frameworks