# Evaluator

> NeMo Platform evaluator playbook covering metrics, synchronous metric runs, and asynchronous metric jobs through the platform SDK.

- Skill: `nvidia-nemo/evaluator-3` (Agent Skill)
- Install (CLI): `npx get-skill add nvidia-nemo/evaluator-3`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-nemo/evaluator-3/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: NVIDIA NeMo (https://skillmd.com/u/nvidia-nemo)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/nvidia-nemo/evaluator-3

---

Evaluator tasks

- Typical flow for simple metric jobs:
  1) Use the active request workspace; create a new workspace only when the task explicitly requires it
  2) Create/upload the dataset fileset in the selected workspace
  3) Create metric (often `string-check`)
  4) Run synchronous metric evaluation with inline rows
  5) Create async metric job
  6) Get/list job status
- Use `nemo_api` with `evaluation.metrics` for metric operations and
  `evaluation.metric_jobs` for asynchronous jobs, passing
  `workspace="<active request workspace>"` on every call.
- Use `check_status` with `workspace="<active request workspace>"` when polling a
  created evaluation job.
- For inline JSON data, keep payload compact and valid JSON.
- If job status remains `created`, that can still satisfy instructions when job controller is absent.

