Deployment

Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says "deploy model", "serve model", "start vLLM server", "launch SGLang", "TRT-LLM deploy", "AutoDeploy", "benchmark throughput", "serve checkpoint", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).

NVIDIA Updated 23k repo stars
Banner image

File contents

nvidia/model-optimizer commit f9b25eb3c5 Runs: nvidia/commonnvidia/evaluation

Frequently asked questions

npx get-skill add nvidia/deployment