Edge LLM Benchmark Performance

Benchmark TensorRT Edge-LLM serving performance (TTFT, inter-token latency, request latency, output throughput, concurrency scaling) through the OpenAI-compatible server with AIPerf. Use for "benchmark performance", "measure TTFT/ITL/tokens per second", prefill or decode sweeps over input and output lengths, in-flight-batching concurrency scaling, or comparing two deployments (quantization, speculative decoding, engine profile, another runtime). Do not use to launch a server only (edge-llm-serve), to score dataset accuracy (edge-llm-benchmark-accuracy), or to profile kernels.

NVIDIA Updated 23k repo stars

File contents

nvidia/tensorrt-edge-llm commit cf75a27f48

Frequently asked questions

npx get-skill add nvidia/edge-llm-benchmark-performance