Modelopt Deployment

Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says "deploy model", "serve model", "start vLLM server", "launch SGLang", "TRT-LLM deploy", "AutoDeploy", "benchmark throughput", "serve checkpoint", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use modelopt-ptq) or evaluating accuracy (use modelopt-evaluation).

NVIDIA Updated 23k repo stars
Banner image

File contents

NVIDIA/Model-Optimizer commit 22ce5900bf Runs: nvidia/commonnvidia/evaluation

Frequently asked questions

npx get-skill add nvidia/modelopt-deployment