Edge LLM Serve

Launch and verify TensorRT Edge-LLM's experimental OpenAI-compatible server for one Hugging Face checkpoint or local checkpoint. Use for `trtedge serve`, OpenAI API requests, server sizing, streaming, tools, media, batching, speculative decoding, or a server startup failure. Do not use for ONNX export or C++ inference (edge-llm-convert-model), installation (edge-llm-install), or adding architecture support (edge-llm-add-model).

NVIDIA Updated 23k repo stars

File contents

nvidia/tensorrt-edge-llm commit c6ea8859fa

Frequently asked questions

npx get-skill add nvidia/edge-llm-serve