Edge LLM Benchmark Accuracy

Measure dataset accuracy (MMLU and other NeMo Evaluator benchmarks) of a TensorRT Edge-LLM deployment through its OpenAI-compatible chat endpoint with NeMo Evaluator. Use for "benchmark accuracy", "run MMLU/GSM8K/IFEval on this checkpoint", comparing a quantization, speculative-decoding, or engine-profile change against a baseline score, or gating a change on a minimum score. Do not use to launch a server only (edge-llm-serve), to measure latency or throughput (edge-llm-benchmark-performance), or to locate which layer diverges after a mismatch (edge-llm-few-layer-test, edge-llm-accuracy-debugging).

NVIDIA Updated 23k repo stars

File contents

nvidia/tensorrt-edge-llm commit 756b8616d0

Frequently asked questions

npx get-skill add nvidia/edge-llm-benchmark-accuracy