Eval Author
Work on repository-owned evaluation suites and understand agent traces. Route
each request to the narrow sub-flow that owns it.
A report that a downstream model trusts has to be right. A plausible report is
worse than no report when somebody acts on it.
The standard
Every fact you record comes from authoritative evidence, not a guess.
The authority depends on the sub-flow:
- For suite discovery, Harbor's validators judge runnability. A file's presence
doesn't prove that Harbor accepts it.
- For audit-spec validation, the bundled schema and validator judge the finite
audit.md coverage denominator.
- For task creation, Harbor's Oracle judges task solvability and verifier
correctness; measured ATIF proves whether repeated runs close the selected gap.
- For trace inspection, Intake establishes what happened. Local source code can
explain behavior, but it can't replace recorded trace evidence.
- For trace-derived environments, canonical ATIF establishes the request and
Harbor's NOP and Oracle runs prove the generated environment.
No sub-flow reimplements a provider's rules. When evidence can't settle a claim,
the report marks the claim unproven or uncertain.
Vocabulary
The sub-flows share this language, and reports use it verbatim.
| Term |
Meaning |
| Check |
One named result: pass, warn, or fail. Carries a message and, when it fails, a hint |
| Required |
A failing required check blocks the suite. Report the suite as not ready |
| Advisory |
A warning worth surfacing that blocks nothing |
| Rung |
One step of a provider's validation ladder, ordered so a lower rung's failure often clears once a higher one is fixed |
| Proven |
A provider judged this check. An unproven check is an observation and never evidence |
| Provider |
The evaluation framework that owns the rules. Harbor today |
| Finding |
One trace claim categorized as behavior, issue, recovery, or uncertainty, with evidence IDs |
| Outcome |
The trace assessment: success, failure, or unknown |
Sub-flows
Read the sub-flow's own SKILL.md and follow it. This file carries the standard
and the boundaries; the sub-flow carries the steps.
| Sub-flow |
Use it to |
eval-author-discover |
Establish whether a repository's evaluations run, name the rung that fails, and get the exact command to run them |
eval-author-audit |
Generate and validate a finite audit.md coverage denominator, write per-method coverage/details files for one ATIF trace, then aggregate coverage reports |
eval-author-task-create |
Create one Harbor-native task from one actionable uncovered tool, prove it with Oracle, and accept it only when repeated measured runs close the gap |
eval-author-inspect-trace |
Understand one Intake trace without presuming that the trace contains a failure. Not user-invocable; this skill selects it |
eval-author-trace-environment |
Normalize one trace to ATIF, make a privacy-reviewed candidate decision, and build a private Harbor task when evidence supports it |
eval-author-audit works one level above tasks: it generates and validates the
coverage denominator, measures traces against it, and aggregates deterministic
coverage reports. eval-author-task-create consumes only actionable tool gaps
from that report and uses Harbor's native task scaffolder rather than guessing a
task layout.
Boundaries
These hold for every sub-flow. They exist because the repository belongs to the
user, not to you.
- Propose, never mutate customer source. Read the user's source and report on
it. Do not edit, move, or reformat any of it, including its
.gitignore. The
only files you add belong under .eval-author/, which is theirs to commit or
ignore.
eval-author-discover scripts write nothing; eval-author-audit writes only
requested audit artifacts; eval-author-task-create writes only drafts,
proposals, job outputs, and measurements there; eval-author-trace-environment
writes only private, gitignored task workspaces there.
- A missing tool is a finding, not a task. When the provider is not installed,
say so and stop short of proving anything. Report what you found regardless, and
do not install the provider into the user's environment.
- Do not run without approval. Discovery proves an existing suite can run and
hands over the command. Task creation may run Oracle locally, then starts
real-agent jobs only when the user explicitly asked for or approved that spend.
- Trusted repositories only. Validating a config can execute repository code,
because an agent named by import path gets imported. If the repository is not
trusted, say so and stop.
- Intake reads are narrow. Only
eval-author-inspect-trace reads Intake. It
uses read-only nemo intake commands against the configured instance and
workspace. No sub-flow discovers accounts, ingests data, uploads files, or
changes a remote resource.
Reporting
Lead with the verdict or outcome, then the evidence.
State whether the findings are proven, whether the suite is ready, and the names
of the checks that failed. Never describe a suite as ready while a required check
fails, and never present an observation as proof. When a sub-flow could not reach
its provider, the only honest headline is that nothing was proven.
For a trace, use success, failure, or unknown. Tie key moments and findings
to span IDs, evaluator result IDs, or source symbols. A healthy trace doesn't
need an issue finding.
1---2name: eval-author3description: Work on evaluation suites in a user's repository, derive an environment from trace evidence, or understand an agent run from NeMo Intake. Owns the evidence standard that every Eval Author sub-flow follows. Use when the user asks "help me with my evals", "what's the state of the eval suite here?", "what happened in this trace?", or when you need to pick between the Eval Author sub-flows. Routes to a sub-flow and changes none of the user's source. The selected sub-flow uses the provider's supported tools and saves its findings under `.eval-author/`.4license: Apache-2.05---67# Eval Author89Work on repository-owned evaluation suites and understand agent traces. Route10each request to the narrow sub-flow that owns it.1112A report that a downstream model trusts has to be right. A plausible report is13worse than no report when somebody acts on it.1415## The standard1617**Every fact you record comes from authoritative evidence, not a guess.**1819The authority depends on the sub-flow:2021- For suite discovery, Harbor's validators judge runnability. A file's presence22 doesn't prove that Harbor accepts it.23- For audit-spec validation, the bundled schema and validator judge the finite24 `audit.md` coverage denominator.25- For task creation, Harbor's Oracle judges task solvability and verifier26 correctness; measured ATIF proves whether repeated runs close the selected gap.27- For trace inspection, Intake establishes what happened. Local source code can28 explain behavior, but it can't replace recorded trace evidence.29- For trace-derived environments, canonical ATIF establishes the request and30 Harbor's NOP and Oracle runs prove the generated environment.3132No sub-flow reimplements a provider's rules. When evidence can't settle a claim,33the report marks the claim unproven or uncertain.3435## Vocabulary3637The sub-flows share this language, and reports use it verbatim.3839| Term | Meaning |40|---|---|41| Check | One named result: `pass`, `warn`, or `fail`. Carries a message and, when it fails, a hint |42| Required | A failing required check blocks the suite. Report the suite as not ready |43| Advisory | A warning worth surfacing that blocks nothing |44| Rung | One step of a provider's validation ladder, ordered so a lower rung's failure often clears once a higher one is fixed |45| Proven | A provider judged this check. An unproven check is an observation and never evidence |46| Provider | The evaluation framework that owns the rules. Harbor today |47| Finding | One trace claim categorized as `behavior`, `issue`, `recovery`, or `uncertainty`, with evidence IDs |48| Outcome | The trace assessment: `success`, `failure`, or `unknown` |4950## Sub-flows5152Read the sub-flow's own `SKILL.md` and follow it. This file carries the standard53and the boundaries; the sub-flow carries the steps.5455| Sub-flow | Use it to |56|---|---|57| `eval-author-discover` | Establish whether a repository's evaluations run, name the rung that fails, and get the exact command to run them |58| `eval-author-audit` | Generate and validate a finite `audit.md` coverage denominator, write per-method coverage/details files for one ATIF trace, then aggregate coverage reports |59| `eval-author-task-create` | Create one Harbor-native task from one actionable uncovered tool, prove it with Oracle, and accept it only when repeated measured runs close the gap |60| `eval-author-inspect-trace` | Understand one Intake trace without presuming that the trace contains a failure. Not user-invocable; this skill selects it |61| `eval-author-trace-environment` | Normalize one trace to ATIF, make a privacy-reviewed candidate decision, and build a private Harbor task when evidence supports it |6263`eval-author-audit` works one level above tasks: it generates and validates the64coverage denominator, measures traces against it, and aggregates deterministic65coverage reports. `eval-author-task-create` consumes only actionable tool gaps66from that report and uses Harbor's native task scaffolder rather than guessing a67task layout.6869## Boundaries7071These hold for every sub-flow. They exist because the repository belongs to the72user, not to you.7374- **Propose, never mutate customer source.** Read the user's source and report on75 it. Do not edit, move, or reformat any of it, including its `.gitignore`. The76 only files you add belong under `.eval-author/`, which is theirs to commit or77 ignore.78 `eval-author-discover` scripts write nothing; `eval-author-audit` writes only79 requested audit artifacts; `eval-author-task-create` writes only drafts,80 proposals, job outputs, and measurements there; `eval-author-trace-environment`81 writes only private, gitignored task workspaces there.82- **A missing tool is a finding, not a task.** When the provider is not installed,83 say so and stop short of proving anything. Report what you found regardless, and84 do not install the provider into the user's environment.85- **Do not run without approval.** Discovery proves an existing suite can run and86 hands over the command. Task creation may run Oracle locally, then starts87 real-agent jobs only when the user explicitly asked for or approved that spend.88- **Trusted repositories only.** Validating a config can execute repository code,89 because an agent named by import path gets imported. If the repository is not90 trusted, say so and stop.91- **Intake reads are narrow.** Only `eval-author-inspect-trace` reads Intake. It92 uses read-only `nemo intake` commands against the configured instance and93 workspace. No sub-flow discovers accounts, ingests data, uploads files, or94 changes a remote resource.9596## Reporting9798Lead with the verdict or outcome, then the evidence.99100State whether the findings are proven, whether the suite is ready, and the names101of the checks that failed. Never describe a suite as ready while a required check102fails, and never present an observation as proof. When a sub-flow could not reach103its provider, the only honest headline is that nothing was proven.104105For a trace, use `success`, `failure`, or `unknown`. Tie key moments and findings106to span IDs, evaluator result IDs, or source symbols. A healthy trace doesn't107need an issue finding.
NVIDIA NeMo