Eval Author Discover

Eval Author: discover

The Eval Author discovery pass. Read eval-author for the standard this follows, the shared vocabulary, and the boundaries that apply throughout. In short: Harbor judges every fact recorded here, and anything Harbor did not judge is marked unproven and is not evidence.

Three phases, in order. The bundled script runs all three in one invocation.

1. Probe. Is Harbor importable by this interpreter?
2. Inventory. Which config files, datasets, and task directories does the repository own?
3. Judge. Only when Harbor is importable: run Harbor's full validation ladder.

Without Harbor, phase 3 cannot run and no claim about runnability is possible. The report still comes back, every finding marked unproven, with a required failure naming what to install.

Before you start

Run the script with the interpreter that has Harbor installed. This is the one step people get wrong, and getting it wrong voids the whole report.

A harbor command on your PATH does not mean Harbor is importable by the Python you are about to run. A repository with its own virtual environment usually needs that environment's interpreter. Try these in order until one prints a version:

for py in .venv/bin/python ./venv/bin/python python3; do
  "$py" -c "import harbor, sys; print(sys.executable, harbor.__version__)" 2>/dev/null && break
done

Nothing prints a version when Harbor is not installed anywhere. Do not install it yourself; in the user's repository the missing environment is the finding. Tell them what you found and ask how they want to proceed.

The report records which mode produced it either way, in runtime.harbor_importable and the top-level proven field.

Step 1: run discovery

Point the script at the repository root, not at a suite directory. It searches for configs to a depth of four directories and finds datasets at any depth.

.venv/bin/python <skill_dir>/scripts/discover.py --repo .

One JSON object goes to stdout, and --compact puts it on one line. The script writes no files; you save the report in Step 5.

The exit code carries the verdict, so check it:

• 0 — every repository-owned config passed every required check
• 1 — a required check failed, Harbor was unavailable, or the path was unusable

Only run this against a repository you trust. Validating a config that names an agent import_path imports that module, which executes its top-level code.

Step 2: read the verdict

Read these four fields before any others.

run_command is deliberately absent when several configs exist. Picking one for the user guesses at intent, so ask which suite they mean and build the command from that config's path.

Step 3: fix what failed

Each check names one rung of Harbor's ladder. Work top to bottom, because a lower rung's failure often disappears once you fix a higher one.

Step 4: verify before you report

Discovery changes none of the user's source, so verification means confirming the report describes the repository they meant:

1. proven is true. When it is false, report only that Harbor is missing.
2. repo_root is the repository they named.
3. configs lists the suite they care about. An empty list on a repository they described as having evals means the configs sit deeper than four directories, or declare no datasets or tasks list.
4. task_count is in the range they expect. A count of zero with a passing tasks check means the config resolves tasks from a registry, not from disk. Report proven, runnable, and the failing check names. Never describe a suite as ready to run while runnable is false.

Step 5: save the report

Write the report to .eval-author/discovery.md, so the next model and the user's teammates inherit the findings instead of rerunning discovery to get them back. Lead with the JSON as front matter, verbatim, then the verdict, the failing checks by name, and the run command. Never paraphrase a check; its wording is the evidence.

Leave the file in the working tree and say where it is. Committing it is the user's call, and worth suggesting. Do not touch their .gitignore. A rerun replaces the file rather than merging into it.

Files in this skill

Provider-specific code sits under scripts/providers/, so support for a second evaluation provider is an added directory rather than a change to the entry point.

The provider directory deliberately sits one level down. A scripts/harbor/ directory would be importable as harbor, which on a machine without Harbor makes find_spec("harbor") succeed and the probe report an install that is not there.