NeMo Helix Evals
Use this skill to turn a domain-specific benchmark and expert-written rubric
into a reproducible evaluation. The agent should parse the rubric, choose the
simplest correct composable primitive for each criterion, generate
human-reviewable config/artifacts, run the evaluation, and explain both scores
and reasoning.
Use the NeMo Helix Evals SDK as the source of truth for metric names, fields,
templates, execution modes, result shapes, and failure behavior. Keep this skill
focused on SDK guidance. Use CLI commands only when the user explicitly needs a
remote platform job or an existing platform resource.
Runtime Context
This skill can run in two contexts:
- In the NeMo Helix repo, expect the SDK to be importable through the repo
environment, for example with
uv run commands from the repo root. Use the
repo-relative SDK reference paths in references/sdk-execution.md when
checking exact schemas or tests.
- As an installed skill outside the repo, do not assume the source tree is
present. Use installed Python imports, package metadata, CLI help, or
user-provided files first; treat repo paths as developer fallback references.
Prefer local SDK runs for iteration and smoke tests. Use remote platform jobs
only when the user needs platform-managed execution, existing platform
resources, or a job result from a deployed environment. For local SDK runs,
api_key_secret resolves through the local environment; for remote jobs, the
same name should refer to a platform secret in the job workspace.
Reference Files
Load these files only when the task needs the detail:
- Metric selection:
references/metric-selection.md
- Read when mapping rubric criteria to SDK metrics, choosing deterministic vs
LLM judges, or working with RAG/agentic/tool-calling metrics.
- SDK execution:
references/sdk-execution.md
- Read before guessing a metric schema, execution parameter, online/offline
setup, parser, template, or SDK result shape.
- Benchmark reproduction:
references/benchmark-reproduction.md
- Read for BYOB protocol, long-running benchmark reproduction, artifact
contracts, and external-safe output requirements.
- Troubleshooting:
references/troubleshooting.md
- Read when SDK runs fail, judge outputs do not parse, provider calls time
out, or generated artifacts need recovery.
Default Loop
- Identify the BYOB inputs: questions, rubric criteria, responses or live
model endpoints, weights, pass threshold, and desired summary breakdowns.
- Clarify the evaluation target: model output quality, judge quality, RAG
quality, tool use, benchmark regression, or bring-your-own benchmark
reproduction.
- Judge-quality evaluation checks whether a judge agrees with labels for
existing responses.
- Generator-quality evaluation creates fresh responses and scores those
responses with a fixed judge or deterministic checker.
- Choose the simplest correct SDK metric class or composed set of classes for
each criterion. See
references/metric-selection.md.
- Generate a small, human-reviewable config or code artifact instead of
burying criterion logic in one-off scripts.
- Build a tiny inline dataset with at least one expected pass and one expected
fail.
- Run locally with
Evaluator().run_sync(...) or await Evaluator().run(...).
See references/sdk-execution.md.
- Inspect
result.print_summary(), row_scores, and aggregate_scores.
- When errors arise, fix dataset shape, Jinja templates, parser paths,
prompts, model config, or secrets. See
references/troubleshooting.md.
- Move to remote platform jobs only after the local SDK run behaves as
expected.
Do not stop at "the score is bad." Explain what failed and what evidence
supports that conclusion.
Core Selection Rules
Prefer deterministic primitives before LLM calls. Use LLM judges for nuance,
not for checks that can be expressed as exact, string, numeric, or code logic.
Use online generation only when the evaluator should call a model or agent
before scoring. Pass target=Model(...) or target=Agent(...) and a
prompt_template; otherwise keep evaluation offline and put outputs in the
dataset rows.
For external, custom, or bring-your-own benchmarks, keep the protocol explicit
and reproducible.
Separate judge-quality evaluation from generation-quality evaluation. If the
target is report generation, generate fresh model responses, score them with a
fixed judge, and aggregate fulfilment. Do not treat human labels for existing
baseline responses as labels for new model outputs.
For public or broadly shared skills and reports, avoid internal endpoint names,
internal model IDs, authentication details, and secret paths. Use placeholders
such as <provider-base-url>, <generator-model-id>, <judge-model-id>, and
<secret-reference>. Report parameter categories and aliases instead of
revealing private infrastructure.
Completion Criteria
A good evaluator answer includes:
- The evaluation goal and chosen SDK metric class or benchmark shape.
- The dataset shape and any templates or parser paths.
- The exact SDK snippet or evaluation spec used.
- Row-level and aggregate evidence for local runs, or job status/result
evidence for remote runs.
- Any caveats about reproducibility, provider config, or calibration.
1---2# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.3# SPDX-License-Identifier: Apache-2.045name: nemo-evals6description: >7 NeMo Helix Evals SDK-first rubric-to-eval guide for BYOB (Bring Your Own8 Benchmark): parse domain expert rubrics, choose composable evaluation9 primitives, generate human-reviewable eval configs/artifacts, and run10 reproducible exact, numeric, LLM-as-judge, code/custom, composite,11 RAG/agentic, and tool-calling evaluations. Use when a task involves12 questions/rubrics/responses files, rubric criteria, benchmark scoring,13 evaluator primitive selection, or reusable evaluation artifacts.14preconditions:15 - evaluator_sdk_available16compatibility: Designed for installed NeMo Helix skill use; repo-relative SDK paths are developer fallbacks when a checkout is available.17allowed-tools: Bash, Read18metadata:19 source: https://skillmd.com/skills/nvidia-nemo/nemo-evals20 author: NeMo Helix Team <nemo-helix@nvidia.com>21 user-invocable: true22---2324# NeMo Helix Evals2526Use this skill to turn a domain-specific benchmark and expert-written rubric27into a reproducible evaluation. The agent should parse the rubric, choose the28simplest correct composable primitive for each criterion, generate29human-reviewable config/artifacts, run the evaluation, and explain both scores30and reasoning.3132Use the NeMo Helix Evals SDK as the source of truth for metric names, fields,33templates, execution modes, result shapes, and failure behavior. Keep this skill34focused on SDK guidance. Use CLI commands only when the user explicitly needs a35remote platform job or an existing platform resource.3637## Runtime Context3839This skill can run in two contexts:4041- In the NeMo Helix repo, expect the SDK to be importable through the repo42 environment, for example with `uv run` commands from the repo root. Use the43 repo-relative SDK reference paths in `references/sdk-execution.md` when44 checking exact schemas or tests.45- As an installed skill outside the repo, do not assume the source tree is46 present. Use installed Python imports, package metadata, CLI help, or47 user-provided files first; treat repo paths as developer fallback references.4849Prefer local SDK runs for iteration and smoke tests. Use remote platform jobs50only when the user needs platform-managed execution, existing platform51resources, or a job result from a deployed environment. For local SDK runs,52`api_key_secret` resolves through the local environment; for remote jobs, the53same name should refer to a platform secret in the job workspace.5455## Reference Files5657Load these files only when the task needs the detail:5859- Metric selection: `references/metric-selection.md`60 - Read when mapping rubric criteria to SDK metrics, choosing deterministic vs61 LLM judges, or working with RAG/agentic/tool-calling metrics.62- SDK execution: `references/sdk-execution.md`63 - Read before guessing a metric schema, execution parameter, online/offline64 setup, parser, template, or SDK result shape.65- Benchmark reproduction: `references/benchmark-reproduction.md`66 - Read for BYOB protocol, long-running benchmark reproduction, artifact67 contracts, and external-safe output requirements.68- Troubleshooting: `references/troubleshooting.md`69 - Read when SDK runs fail, judge outputs do not parse, provider calls time70 out, or generated artifacts need recovery.7172## Default Loop73741. Identify the BYOB inputs: questions, rubric criteria, responses or live75 model endpoints, weights, pass threshold, and desired summary breakdowns.762. Clarify the evaluation target: model output quality, judge quality, RAG77 quality, tool use, benchmark regression, or bring-your-own benchmark78 reproduction.79 - Judge-quality evaluation checks whether a judge agrees with labels for80 existing responses.81 - Generator-quality evaluation creates fresh responses and scores those82 responses with a fixed judge or deterministic checker.833. Choose the simplest correct SDK metric class or composed set of classes for84 each criterion. See `references/metric-selection.md`.854. Generate a small, human-reviewable config or code artifact instead of86 burying criterion logic in one-off scripts.875. Build a tiny inline dataset with at least one expected pass and one expected88 fail.896. Run locally with `Evaluator().run_sync(...)` or `await Evaluator().run(...)`.90 See `references/sdk-execution.md`.917. Inspect `result.print_summary()`, `row_scores`, and `aggregate_scores`.928. When errors arise, fix dataset shape, Jinja templates, parser paths,93 prompts, model config, or secrets. See `references/troubleshooting.md`.949. Move to remote platform jobs only after the local SDK run behaves as95 expected.9697Do not stop at "the score is bad." Explain what failed and what evidence98supports that conclusion.99100## Core Selection Rules101102Prefer deterministic primitives before LLM calls. Use LLM judges for nuance,103not for checks that can be expressed as exact, string, numeric, or code logic.104105Use online generation only when the evaluator should call a model or agent106before scoring. Pass `target=Model(...)` or `target=Agent(...)` and a107`prompt_template`; otherwise keep evaluation offline and put outputs in the108dataset rows.109110For external, custom, or bring-your-own benchmarks, keep the protocol explicit111and reproducible.112113Separate judge-quality evaluation from generation-quality evaluation. If the114target is report generation, generate fresh model responses, score them with a115fixed judge, and aggregate fulfilment. Do not treat human labels for existing116baseline responses as labels for new model outputs.117118For public or broadly shared skills and reports, avoid internal endpoint names,119internal model IDs, authentication details, and secret paths. Use placeholders120such as `<provider-base-url>`, `<generator-model-id>`, `<judge-model-id>`, and121`<secret-reference>`. Report parameter categories and aliases instead of122revealing private infrastructure.123124## Completion Criteria125126A good evaluator answer includes:127128- The evaluation goal and chosen SDK metric class or benchmark shape.129- The dataset shape and any templates or parser paths.130- The exact SDK snippet or evaluation spec used.131- Row-level and aggregate evidence for local runs, or job status/result132 evidence for remote runs.133- Any caveats about reproducibility, provider config, or calibration.