Evaluation

Evaluation

Running an environment in this repo — both while developing/iterating on it (smoke-testing changes) and when benchmarking a finished env. Run everything from the repo root, always via uv.

Environments are v1 tasksets: a package that exports a Taskset. Run one with the eval CLI, by taskset id: uv run eval <taskset-id>.

Setup

Editable, local install from the repo root (not from inside the env dir). Envs live under environments/, grouped into semantic group folders (e.g. environments/math/, environments/swe/):

uv pip install -e path/to/env

Editable means code edits are picked up without reinstalling — so during development you can change the env and immediately re-run it. Only re-run this install after editing pyproject.toml (e.g. new deps). v1 tasksets pin a verifiers pre-release; if the install complains, add --prerelease=if-necessary (never the blanket --prerelease=allow, which bumps unrelated deps to betas — e.g. a sympy rc + mpmath alpha that can't import together).

Harbor-based envs (SWE/terminal tasksets) must depend on verifiers[harbor], not plain verifiers — harbor is an optional extra of verifiers, and without it the eval crashes at taskset load with ModuleNotFoundError: No module named 'harbor.models'.

In a dirty worktree, use uv run --no-sync ... after the editable install so machine-local uv settings cannot rewrite the repository's uv.lock. Do not export UV_FROZEN for an eval: subprocess runtimes inherit it, and their unlocked inline uv scripts cannot start in frozen mode.

Run an eval from the repository root, not from inside its virtualenv. NLTK's import protection rejects dependencies installed beneath the current working directory. For a separate temporary venv, activate it by absolute path and run uv run --active --no-project eval ... from outside that venv.

After dependency edits, sync the environment project before testing imports:

uv sync --project path/to/env --all-extras

If a fresh environment import fails because resolved package versions are incompatible, fix the environment's pyproject.toml bounds and sync again. Do not patch generated .venv files.

Smoke test

Run a 3x1 (3 tasks, 1 rollout each) in plain-log mode to confirm the env loads and scores end-to-end. Spanning a few tasks surfaces weird things (bad rows, edge-case prompts, inconsistent scoring) that a single task hides:

uv run --no-sync eval <taskset-id> -n 3 -r 1 --no-rich -v --no-push

--no-rich turns off the live dashboard (which is on by default) for plain logs, -v prints prompts/completions, and --no-push keeps development runs local. This is the inner loop while developing — re-run it after each change to verify the dataset loads, the rollout runs, and the rubric scores as expected. Fix any errors here before scaling up.

Full eval

Run all tasks (omit -n) and pick -r so the total sample is representative — usually >500 total rollouts (num_tasks × r) is a good target. For large datasets -r 1 may already clear that; for small ones, bump -r.

Recommended full-eval invocation:

uv run eval <taskset-id> -r <r> -c <N> --no-rich

The run is always saved to disk (see Inspect output) — there's no save flag.

Key flags (eval)

Most tasksets yield records as they are consumed. In Verifiers 0.3.1, -n N uses a lazy head before the runner collects those N tasks, but automationbench_env, general_agent, tau2_bench, and tau3_bench still construct their complete task lists. -s calls Taskset.shuffle() first, which must materialize the complete finite taskset, so avoid -s when a smoke test should construct only a prefix.

Configuring the env

Pass typed, dotted flags or a TOML file — there is no -a JSON blob:

uv run eval wikispeedia --env.taskset.min-path-length 5 --env.taskset.max-path-length 8
uv run eval <taskset-id> @ eval.toml          # a launch TOML is re-runnable

Common knobs: --env.agent.max-turns, --env.agent.max-total-tokens, --sampling.max-tokens, --sampling.temperature, --env.agent.harness.id, --env.agent.runtime.type. uv run eval <taskset-id> -h prints the full typed help, narrowed to the chosen taskset/harness.

TasksetConfig.system_prompt is reserved for --env.taskset.system-prompt, a path to a prompt file that overrides every task during iteration. Environment configs must not redefine it as inline text; use a distinct field such as task_system_prompt for an inline/default task prompt.

List-valued flags on union configs (e.g. --env.agent.runtime.labels on the prime runtime) must be passed as a space-separated JSON value — --env.agent.runtime.labels '["ci"]'. The bare form (... labels ci) and the =-form with JSON both parse as a string and fail validation with Input should be a valid list.

A dotted override below a nested RuntimeConfig can replace that whole runtime with the selected provider's defaults. After overriding any nested runtime field, inspect configs/resolved/eval.json and confirm its image, workdir, runtime type, and resources before treating the smoke test as representative.

Harness & runtime (sandboxed / agentic tasksets)

A harness belongs to an agent seat: on the default single-agent env the seat is agent (--env.agent.harness.*); a multi-agent env names its own seats (e.g. --env.solver.harness.* and --env.judge.harness.* on agentic-judge). A taskset that bundles its own harness runs with it by default; otherwise pass --env.agent.harness.id. Select where rollouts execute with --env.agent.runtime.type:

In Verifiers 0.3.1, the execution runtime belongs to the environment's agent role, not HarnessConfig. If a bundled harness must run with the interpreter that installed its package, export a custom Env/EnvConfig and set its agent default to AgentConfig(runtime=SubprocessConfig()).

Toolsets running on Docker or Prime runtimes require Verifiers 0.3.1 to be installed from its source checkout, whether or not they are colocated with the harness. The framework builds and uploads its own source distribution with the environment package; a wheel-only Verifiers install has no pyproject.toml to build and fails before the tool server starts.

Prime port exposure can return before its public route is reachable from another Prime sandbox. A taskset that supports a separate Prime tool-server runtime must check the exposed route from Prime before handing it to the agent; a localhost probe inside the server only proves that the process is listening.

TaskData.network_allow=[] is the Verifiers 0.3.1 default-deny policy for Docker and Prime solver runtimes; Prime requires vm=true. Verifiers applies it after trusted task setup and retains its own interception and MCP routes. Subprocess and Modal do not implement task-data network policies, although a task-owned Modal runtime can set network_access=false. A secondary runtime created directly with make_runtime() must use a restricted config and call prepare_execution(...) after setup, before it runs agent-controlled code.

uv run eval <taskset-id> --env.agent.runtime.type subprocess  # local process
uv run eval <taskset-id> --env.agent.runtime.type docker      # local container (needs local docker)
uv run eval <taskset-id> --env.agent.runtime.type prime       # remote prime VM (default; needs auth)
uv run eval <taskset-id> --env.agent.runtime.type modal       # remote modal sandbox (needs auth)

Harbor tasksets should reuse HarborTask for environment upload and packaged test.sh scoring. When gated verifier inputs must not enter the agent box, export a HarborEnv subclass, set HarborData.verifier, and collect only the solution artifacts that the separate verifier needs.

For packages whose task Dockerfile only selects a base image and copies task assets, resolve the FROM image, set upload_environment=True, and let HarborTask.setup stage the assets; Verifiers deliberately does not build task Dockerfiles.

Harbor can also load an unpublished dataset directly from Git. Set HarborConfig.dataset to "" and repo to a pinned GitHub URL of the form https://github.com/<org>/<repo>/tree/<commit>/<tasks-subdir>; the implicit repository loader discovers the task directories, downloads them together, and caches the export. Reuse HarborTaskset.load() instead of implementing another archive downloader.

RewardKit judges routed through an OpenAI-compatible endpoint need LiteLLM's provider prefix in front of the endpoint model id: a pinference model id such as openai/gpt-5.5 is passed as openai/openai/gpt-5.5. Pass that model, the endpoint, and a credential template through HarborData.verifier_env; the packaged RewardKit verifier does not consume Verifiers judge sampling settings. Configure task-owned judge endpoints under --env.taskset.task.judge.*.

Prime containers accept explicit registry refs such as GHCR, but Prime VM image builds require Docker Hub or Prime registry refs; point VM smoke tests at a published Prime mirror.

The built-in bash harness (a bash + edit agent) is the fallback when no harness id is given; null is a tool-less chat loop (MCP tools only). For "bring your own search" tasksets that ship no search tool of their own (e.g. openseeker, redsearcher, s1-deepresearch, arxivmath), give the agent a Serper-backed search tool with --env.agent.harness.search true (needs SERPER_API_KEY in the eval environment):

uv run eval openseeker --env.agent.harness.id bash --env.agent.harness.search true --env.agent.runtime.type prime

Concurrency

-c caps episodes in flight (default 128). An episode may contain multiple agent runs; those run one at a time by default. Set --env.max-concurrent-agents <N> only when an environment should play several agents from the same episode concurrently. The upper bound on simultaneous agent runs is therefore roughly -c * --env.max-concurrent-agents.

Raise -c for cheap, non-sandboxed tasksets; keep it lower for sandboxed ones (containers/remote runtimes) and tune up from there. Eval uses the env-server pool by default; its settings live below --serve.pool.* (for example, --serve.pool.type static --serve.pool.num-workers 4), while --serve.max-concurrent optionally caps episodes per worker. Use --no-serve only when in-process execution is intentional.

Models

The taskset's default model is usually fine (deepseek/deepseek-v4-flash). Otherwise pick by tier (-m <slug>):

• Cheap — deepseek/deepseek-v4-flash or z-ai/glm-5.1.
• Good (more capable) — openai/gpt-5.4 or openai/gpt-5.5 at medium reasoning.
• Very hard only — openai/gpt-5.5 is the strongest but expensive. Avoid it by default; use it only as a last resort to confirm a task is solvable when the cheaper models get zero reward.

Live dashboard in tmux

The Rich dashboard is on by default and shows a live reward <mean> · err <share> headline, but it doesn't render in a plain captured shell. To watch it live, run inside tmux and drive it with send-keys:

tmux new-session -d -s eval
tmux send-keys -t eval 'uv run eval <taskset-id> -r <r>' Enter
# watch it:
tmux attach -t eval        # or: tmux capture-pane -t eval -p

For headless/automated runs, prefer --no-rich and follow the logs directly.

Inspect output

Each run is saved to a fresh per-run dir (so runs never overwrite each other):

outputs/<run-dir>/
  configs/eval.toml           # launch config when the run starts from CLI flags
  configs/resolved/eval.json  # complete config; use this to resume
  traces.jsonl                 # one full episode per line
  logs/attempt_<n>/eval.log    # logs for one run or resume attempt
  logs/latest -> attempt_<n>   # current attempt

# find the newest run
ls -dt outputs/*/ | head

# avg reward across rollouts (recomputed — aggregates aren't stored)
jq -s '[.[].traces[] | [.rewards[]? | .score * .weight] | add // 0] | if length > 0 then add / length else 0 end' outputs/<...>/traces.jsonl

The dashboard shows the avg reward live; with --no-rich, recompute it from traces.jsonl (each line is an episode whose traces[] carry the per-reward breakdown). Skim a few episodes for sanity before publishing.