Benchmark dataset accuracy with NeMo Evaluator
NeMo Evaluator scores a deployment through POST /v1/chat/completions, so the
number measures the whole served path: checkpoint, quantization, engine
profile, chat template, sampling, and server. It is a regression signal, not a
proof of numerical equivalence.
Instructions
Take the benchmark, sample budget, and baseline from the user; then run Phases 1 to 5: prepare the evaluator, serve with a sized profile, run the evaluation, validate the run, and compare with the baseline. Report the score, sample count, both commands, and the results path.
Read first
docs/source/user_guide/examples/integration/nemo-evaluator.mdscripts/run_nemo_eval.py(--helpand the parser at its end)requirements-nemo-evaluator.txt(pinned evaluator packages)tests/model_configs/l0/qwen3_5/qwen3_5_0_8b.yml, casenemo-evaluator: the CI-verified recipe and itsscore.mingate
Inputs
| Input | Required | Notes |
|---|---|---|
| Checkpoint | yes | Exact Hugging Face ID and revision, or absolute local path |
| Benchmark | yes | --eval-type registered by the installed evaluator packages; default to mmlu only if the user has no preference |
| Sample budget | yes | limit_samples for a smoke gate, omitted for a full run; state which |
| Baseline | for comparisons | Score from the reference deployment on the same benchmark, samples, and sampling |
| Gate | optional | --min-score from an agreed baseline; never invent one |
Stop and ask when the benchmark, sample budget, or baseline for a comparison is missing. Do not substitute another benchmark.
Phase 1: environment
- Run
scripts/run_nemo_eval.pywith the Edge-LLM runtime interpreter (fromedge-llm-install): with--modelit starts the server with the current interpreter, which must be able to importtensorrt_edgellm; an evaluator-only venv cannot. Installrequirements-nemo-evaluator.txtinto that environment (the documented default), or keep the evaluator in a separate venv and pass--nemo-evaluator-bin <venv>/bin/nemo-evaluator; do not activate the evaluator venv for the helper. - If a run fails on an unknown option, check
nemo-evaluator run_eval --helpagainst the installed version and report the versions. - The helper has no API-key support. For a server started with
--api-key, callnemo-evaluator run_eval --api_key_name <ENV_VAR>directly. - Confirm the benchmark is registered (
nemo-evaluator ls | grep -i <name>; the full list spans several harnesses) and its dataset is reachable from this host (network or a pre-populated cache). Datasets load through Hugging Facedatasets; a PythonCERTIFICATE_VERIFY_FAILEDor "Couldn't reach ... on the Hub" whilecurlworks means the venv lacks the site CA bundle: exportSSL_CERT_FILEandREQUESTS_CA_BUNDLEto the system bundle (for example/etc/pki/tls/certs/ca-bundle.crt); set them before the first run on such a host. An offline host needs the user's dataset location; do not substitute another dataset. The first run downloads the dataset (MMLU loads 57 subject sets), so allow minutes before the first request reaches the server. - NeMo Evaluator prints telemetry notices; set
NEMO_EVALUATOR_TELEMETRY_LEVEL=0when the user's policy requires it, and say so in the report.
Phase 2: serve
Launch with edge-llm-serve, sized for the benchmark rather than the default:
--max-input-lenmust cover the longest prompt, including few-shot examples.--max-kv-cache-capacitymust cover prompt plusmax_new_tokens.- MMLU answers step by step, so use a generous
max_new_tokens(the CI recipe uses 2048) or answers are cut off before the final choice and scored wrong. - Set the evaluator request timeout above the worst-case generation time (CI uses 300 s); a timeout is retried and inflates the server's request count.
- Use
--parallelism 1for a deterministic baseline. In-flight batching only works with a server you launch yourself (the helper has no passthrough for it). - Add
--speculative-config, a quantized checkpoint, or any profile change only when that is the variable under test.
run_nemo_eval.py --model <checkpoint> can start and stop the server itself
(see its --help; its profile defaults differ from the server's, and the served
name is the checkpoint's basename). Use it for a one-shot run; start the server
yourself (edge-llm-serve) to reuse one engine for several benchmarks.
Phase 3: run
Run from the repository root with Python 3.10 or newer (the helper is a
repository script). A first MMLU run takes minutes (dataset download plus
scoring) and prints many progress lines, so run it in the background or with a
long timeout and redirect output to a file. Dataset caches follow HF_HOME.
Against a server that is already ready:
python scripts/run_nemo_eval.py \
--model-url http://127.0.0.1:<port>/v1/chat/completions \
--eval-type <benchmark> \
--limit-samples <n> --parallelism 1 --max-new-tokens <n> \
--temperature 0.0 --top-p 1.0 \
--output-dir <task-dir>/nemo-results [--min-score <baseline-gate>]
The helper discovers the served model ID from /v1/models. The direct
equivalent is nemo-evaluator run_eval --eval_type ... --model_id ... --model_url ... --model_type chat --output_dir ... --overrides "config.params.limit_samples=...,config.params.parallelism=1".
Use greedy sampling for regression comparisons. Keep benchmark, sample count,
seed, sampling, max_new_tokens, and chat template identical between runs
being compared; otherwise the difference is not attributable.
Phase 4: validate the run
A score alone is not evidence. Before reporting:
- The command exited 0 and printed
NeMo Evaluator score: <value>; "Could not find score key" means the metric name differs. Inspect the JSON under--output-dirand pass the right--score-key; do not report a guess. - The server received one chat request per sample (count them in the server
log, or the
/healthcounters under in-flight batching). Fewer thanlimit_samplesmeans failed or skipped requests. The printed summary has no sample count; the requested count islimit_samplesinresults.yml(the generated--first_n). If you cannot observe the server, say this check was not possible instead of inferring it. - No request errors, timeouts, or 4xx/5xx in the server log. A 413 means the
prompt exceeded
--max-input-len; rerun with a larger profile. - Inspect a few raw responses: the "Examples" section of
<output-dir>/<eval-type>/<eval-type>.htmllists prompts and model answers;report.jsonandresults.ymlhold the metrics and the exact command. Empty or truncated answers meanmax_new_tokensis too small or a reasoning parser is cutting output.
A tiny max_new_tokens (for example 16 in a smoke run) truncates MMLU's
step-by-step answers and scores near zero by design; such a run proves the
plumbing only, so label it a smoke run and do not report the score as accuracy.
Small sample budgets are noisy. For a pass-rate score the standard error is
sqrt(p * (1 - p) / n): about 3 points at p = 0.6 and n = 250. Compute it
for the actual score and sample count, and do not call a difference within two
standard errors a regression; rerun with more samples or the full benchmark.
Phase 5: compare
For a baseline comparison run both deployments back to back on the same device
with identical settings and report both scores, sample counts, and the delta
with its sampling noise. A drop larger than noise goes to
edge-llm-few-layer-test and then edge-llm-accuracy-debugging; preserve the
results directory and server log.
Output format
Report the Edge-LLM revision, evaluator versions, checkpoint, the redacted server
command and the evaluator command, benchmark, samples evaluated, sampling and
max_new_tokens, the score with its sampling error, the baseline and delta when
compared, the results and server log paths, and whether the server was stopped.
Limitations
run_nemo_eval.pydrives only thechatmodel type. Other evaluator model types (completions,completions_logprob) are not exercised by this workflow; do not assume log-likelihood benchmarks work.- Scores from the quantized or speculative path differ from the reference precision by design; judge them against an agreed baseline.
- Do not compare scores across evaluator package versions.
Troubleshooting
| Symptom | Cause and action |
|---|---|
| Server exits before health | Read <output-dir>/edgellm_server.log (--model mode) or your own log; use edge-llm-serve failure routing |
Score below threshold |
Real regression or noise; follow Phase 5 before reporting |
| Scores near zero | max_new_tokens too small or a timeout; raise it with the KV capacity |
| HTTP 413 in the server log | Prompt exceeds --max-input-len; relaunch with a larger profile |
| Dataset download fails | If curl reaches the Hub, set the CA variables (Phase 1 step 4); otherwise no network or cache, so ask the user for a dataset location |
Examples
- "Run MMLU on this checkpoint and fail below 0.50": serve the checkpoint, run
scripts/run_nemo_eval.pywith--eval-type mmluand--min-score 0.50, validate samples and responses, report the score with its sampling error. - "Did the quantized checkpoint lose accuracy against FP16?": run both deployments back to back with identical settings and report both scores, the delta, and the noise.