# Nemo Evals Plugin

> Evaluate models, datasets, and agents with the NeMo Helix Evals plugin. Use for metric selection, SDK checks, platform jobs, result retrieval, and Harbor taskset runs on Kubernetes (`nemo evals runs`).

- Skill: `nvidia-nemo/nemo-evals-plugin` (Agent Skill, multi-file: 23 files)
- Install (CLI): `npx get-skill add nvidia-nemo/nemo-evals-plugin`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-nemo/nemo-evals-plugin/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: Apache-2.0
- Author: NVIDIA NeMo (https://skillmd.com/u/nvidia-nemo)
- Updated: 2026-11-12
- Page: https://skillmd.com/skills/nvidia-nemo/nemo-evals-plugin

---


# Evals Plugin

The Plugin CLI entrypoint is `uv run nemo evals`.

## Purpose

Use this skill to choose an evaluation interface and metric, validate a minimal
example, submit a NeMo Helix evaluation job, and retrieve its results.

## Inputs

Establish these inputs before building an evaluation:

- Evaluation interface: [dataset-driven, task-driven, or retrieval-driven evaluation](references/evaluation-shapes.md#difference-summary)
- Execution interface: standalone SDK evaluation or a durable NeMo Helix job.
- Pass/fail dataset examples: the smallest representative pass and failure cases.
- Metrics: the behaviors to score and the template fields they consume.
- Target: no target for offline scoring, or the model, agent, runner, or precomputed trials that produce outputs.

## Instructions

1. Clarify whether the input is [dataset-driven rows](references/evaluation-shapes.md#dataset-driven-evaluation),
   [task-driven agent work](references/evaluation-shapes.md#task-driven-evaluation), or a
   [BEIR retrieval corpus](references/evaluation-shapes.md#retrieval-driven-evaluation).
   If the user wants the full embed/rerank fine-tune recipe, hand off to `nemo-retrieval-recipes`
   and use this skill only to run and debug retrieve-eval.
2. Choose the simplest metric that measures the requested behavior. Prefer deterministic metrics when possible.
3. Build a tiny smoke case with one expected pass and one expected failure. For a Kubernetes taskset run,
   the `oracle` run in [Harbor on Kubernetes](references/harbor-kubernetes.md) is the smoke case.
4. Validate metric behavior with the standalone SDK. Inspect row-level output, aggregate `count` and
   `nan_count`, and coverage before interpreting a mean.
5. Fix field mappings, prompts, parsers, or task definitions before scaling.
6. Submit the platform job only after the input and scoring shape works.

Read [Metric Selection](references/metric-selection.md) before choosing a
metric for a rubric, RAG workflow, or tool-calling evaluation.

## Choose the execution interface

| Need | Interface |
| --- | --- |
| Fast metric iteration without NeMo Helix | `nhx_evals_sdk.Evaluator` |
| Dataset-driven platform job | `evaluator.submit(...)` or `nemo evals evaluate` |
| Multiple inline/stored metric refs in one job | `nemo evals evaluate` with an `EvaluateInputSpec` |
| Task-driven platform job | `evaluator.submit(tasks=..., target=<runner>)` or `nemo evals agent-evaluate` |
| Retrieval-driven platform job | `nemo evals retrieve-eval` |
| Harbor task suite on a Kubernetes deployment: one Job and OpenSandbox sandbox per task | `nemo evals runs submit`; read [Harbor on Kubernetes](references/harbor-kubernetes.md) |
| Reusable platform definitions and result indexes | `evaluator.metrics`, `.tasks`, `.tasksets`, `.eval_results`, `.agent_eval_results` |

Default to the plugin-specific job commands for durable platform evaluation:
`nemo evals evaluate`, `nemo evals agent-evaluate`, and
`nemo evals retrieve-eval`. The old `nemo evals ... run` and
`nemo evals ... submit` CLI verbs are not part of the supported evaluator
job surface. The one exception is `nemo evals runs submit`, the supported path
for Harbor taskset runs on Kubernetes. For fast metric iteration without the
platform, use the standalone `nhx_evals_sdk.Evaluator` instead.

- Read [SDK Execution](references/execution.md) for datasets, targets,
configuration, field mapping, job lifecycle, and custom metric packaging.
- Read [Stored Resources](references/resources.md) for persisted definitions and
result queries.

## Limitations

- `api_key_secret` is an environment-variable name standalone but a NeMo
  Platform secret name on `submit`. See [API Auth](references/api-auth.md).
- HTTP 409 from a submission often means a referenced platform secret is
  missing, not a duplicate job. Read the response body.
- `intent` is grader metadata and is never shown to the agent; only `inputs`
  reaches it.
- Metric templates use `item.*` for dataset rows but `reference.*`, `sample.*`,
  and `inputs.*` in agent evaluation.
- Metric progress can reach 100 percent before the platform job is terminal.
  Always call `job.wait_until_done()` before retrieving results or downloading
  artifacts.

## CLI Interface

### Prerequisites

All commands in this file assume that the shell's working directory is the root
of the NVIDIA-NeMo/nemo-helix repository.

In a NeMo Helix repository checkout, run commands through the workspace:

```bash
# confirms plugin readiness and lists the registered evaluator jobs.
uv run nemo evals info
# lists available metric names; add a metric name to print its schema.
uv run nemo evals metric-types
# next three commands print the dataset-driven, task-driven, and retrieval job input and
# output schemas - can be very large, use with caution to avoid filling up the context window.
uv run nemo evals evaluate explain
uv run nemo evals agent-evaluate explain
uv run nemo evals retrieve-eval explain
```

When the skill and plugin are installed, use the installed `nemo` command
without assuming a repository root or manually activating `.venv`.

Resolve bundled assets relative to this skill directory. In this repository the
canonical path is `skills/nemo-evals-plugin`; an installed skill may live
under a different skills root.

## Bundled assets

| Path | Use |
| --- | --- |
| `assets/specs/exact_match_metric.json` | Two-row offline smoke spec; submit as-is |
| `assets/specs/llm_as_judge.json` | Online generation + judge; local-first (`NVIDIA_API_KEY`) |
| `assets/specs/fabric_agent_eval.json` | Task-driven Fabric runner spec |
| `assets/specs/gym_agent_eval.json` | Task-driven sandboxed Gym FileSet spec |
| `assets/specs/harbor_taskset_run_openhands.json` | `nemo evals runs submit --spec-file` request: openhands on a gateway model |
| `assets/examples/plugin_sdk_examples.py` | Copyable SDK snippets for each plugin surface |

## Available Scripts

| Script | Purpose | Arguments |
| --- | --- | --- |
| `scripts/generate_example_specs.py` | Generate or drift-check bundled specs | `--check`, `--write` |

In this repository, NeMo uses the displayed workspace command:

```bash
uv run --frozen python skills/nemo-evals-plugin/scripts/generate_example_specs.py --check
```

Do not assume a client-specific `run_script()` helper; use the displayed
`uv run` command.

## Examples

### Dataset-driven evaluation examples

- Follow [Validate standalone, then submit to the platform](references/execution.md#validate-standalone-then-submit-to-the-platform).
  for the two-row pass/fail smoke test and its CLI submission.
- Follow [Map noncanonical fields](references/execution.md#map-noncanonical-fields)
  when dataset columns need `field_mapping`.
- Follow [Getting job results](references/execution.md#getting-job-results)
  for submission, terminal waiting, result retrieval, and artifact download.
- Follow [Store a metric, task, and taskset](references/resources.md#store-a-metric-task-and-taskset)
  for reusable definitions, and [Query persisted results](references/resources.md#query-persisted-results)
  for result lookup.

### Task-driven agent evaluation examples

**Standalone SDK evaluation**

Use `AgentEvaluator().run(...)` for standalone task-driven SDK evaluation. Its
`target` can be a `Model`, a `GenericAgent`, or a direct `AgentTaskRunner`.

**Platform job evaluation**

Use the plugin `agent-evaluate` job for platform task evaluation. Its
target is a `ModelTarget`, `AgentTarget`, `FabricRunnerTarget`,
`HarborRunnerTarget`, or `GymRunnerTarget`; alternatively provide precomputed
`trials`. Provide exactly one of `target` or `trials`.

A `FabricRunnerTarget`'s `source` is an inline config (`{"kind": "fabric",
"source": {"config": {...}}}`) or a registered agent (`nemo agents register`):
`{"kind": "fabric", "source": {"agent": "<name>"}}`. A `HarborRunnerTarget`'s `source`
takes the same registered-agent shape (`{"kind": "harbor", "source": {"agent":
"<name>"}}`) to run it inside each task container, next to a built-in agent
(`{"name": "oracle"}`) or your own (`{"import_path": "pkg:Agent"}`). A `GymRunnerTarget`'s
`source` takes it too (`{"kind": "gym", "source": {"agent": "<name>"}, "resources_server":
"<name>"}`), running it on the sandboxed Gym host with its default model on Gym's policy
server; the job stages a `wheels-v1` package with the harness, so sandboxed execution is required. At submit time the service resolves
the agent exactly as a deployment would — models bound to the workspace
Inference Gateway, no credentials in the spec — and runs it fresh for every
trial; it never calls an existing deployment. An optional `environment` (the
same spec `nemo agents deploy` takes) redirects the agent's declared MCP servers
to mocks, adds process env, and binds secrets by reference. Only
`nemo-agents-spec-v1` agents resolve; a different model is a different agent,
so there is no model override.

Submission accepts inline tasks or a stored `TasksetRef`. Stored tasksets are
resolved in the target workspace.

**Kubernetes taskset run**

To run a Harbor task suite on a Kubernetes deployment, use `nemo evals runs submit`,
not an `agent-evaluate` job. Read [Harbor on Kubernetes](references/harbor-kubernetes.md)
for the oracle smoke run, openhands with a model (closed-book opt-in), Experiments, results and retries.

Read [Agent Evaluation](references/agent-evaluation.md) for inline tasks,
`TasksetRef`, concurrency, fail-fast behavior, sparse-output denominators,
result artifacts, and runner configuration, including Gym FileSet and
sandbox requirements.

### Prepare Fabric in a repository checkout

Fabric runner examples and tests need the optional harness adapters and the
matching Relay gateway:

```bash
uv sync --frozen --package nhx-evals-sdk --extra fabric --inexact
script/dev-install-fabric.sh
```

The install script downloads the checksum-verified `nemo-relay` binary that
matches the locked Python bindings. Add its reported directory to `PATH`, then
use `uv run --frozen --no-sync ...` for Fabric checks so uv does not remove the
optional adapters.

## Output Format

Report a completed platform evaluation in this form:

```text
Job: <job-name>
Status: <terminal-status>
Metrics: <metric-names>
Mean: <aggregate-mean>
Artifacts: <downloaded result or artifact location>
Errors: <error messages>
```

## Read specialized references

- Read [Evaluator API Auth](references/api-auth.md) before using a model,
  agent, remote metric, or durable submission.
- Read [LLM Judge](references/llm-judge.md) before writing judge scores,
  prompts, or parsers.

## Troubleshooting

Read [Evaluator troubleshooting](references/troubleshooting.md) when schema,
authentication, job, result, or runner behavior fails.

## Follow security best practices

Never print, serialize, or commit secret values. Store only environment-variable
names or platform secret references in specs and examples.

