Profile My skills Saved Collections Edit profile Submit a skill

Evaluator

NeMo Platform evaluator playbook covering metrics, synchronous metric runs, and asynchronous metric jobs through the platform SDK.

NVIDIA NeMo 3e6d5af 1005 B Updated
AI & ML

File contents

Evaluator tasks

  • Typical flow for simple metric jobs:
    1. Use the active request workspace; create a new workspace only when the task explicitly requires it
    2. Create/upload the dataset fileset in the selected workspace
    3. Create metric (often string-check)
    4. Run synchronous metric evaluation with inline rows
    5. Create async metric job
    6. Get/list job status
  • Use nemo_api with evaluation.metrics for metric operations and evaluation.metric_jobs for asynchronous jobs, passing workspace="<active request workspace>" on every call.
  • Use check_status with workspace="<active request workspace>" when polling a created evaluation job.
  • For inline JSON data, keep payload compact and valid JSON.
  • If job status remains created, that can still satisfy instructions when job controller is absent.

nvidia-nemo/nemo-platform/tree/main/agents/nemo-studio-assistant-ethos/skills/evaluator commit 3e6d5af044

Browse category

Frequently asked questions

npx skillmds@latest add nvidia-nemo/evaluator-2