# Evaluation > Install and run a verifiers environment — smoke testing during development and full benchmark evals. Covers the `eval` CLI for the tasksets. Use while developing/iterating on an environment, when running/evaluating/benchmarking one, or when the user mentions eval, tasksets, eval runs, or smoke tests. - Skill: `primeintellect-ai/evaluation` (Agent Skill) - Install (CLI): `npx skillmds@latest add primeintellect-ai/evaluation` - Raw SKILL.md: https://api.skillmd.com/api/skills/primeintellect-ai/evaluation/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: primeintellect-ai (https://skillmd.com/u/primeintellect-ai) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/primeintellect-ai/evaluation --- # Evaluation Running an environment in this repo — both while **developing/iterating** on it (smoke-testing changes) and when **benchmarking** a finished env. Run everything from the **repo root**, always via `uv`. Environments are **v1 tasksets**: a package that exports a `Taskset`. Run one with the **`eval`** CLI, by taskset id: `uv run eval `. ## Setup Editable, local install from the repo root (not from inside the env dir). Envs live under `environments/`, grouped into semantic group folders (e.g. `environments/math/`, `environments/swe/`): ```bash uv pip install -e path/to/env ``` Editable means code edits are picked up without reinstalling — so during development you can change the env and immediately re-run it. Only re-run this install after editing `pyproject.toml` (e.g. new deps). v1 tasksets pin a `verifiers` pre-release; if the install complains, add `--prerelease=if-necessary` (never the blanket `--prerelease=allow`, which bumps unrelated deps to betas — e.g. a sympy rc + mpmath alpha that can't import together). Harbor-based envs (SWE/terminal tasksets) must depend on `verifiers[harbor]`, not plain `verifiers` — `harbor` is an optional extra of verifiers, and without it the eval crashes at taskset load with `ModuleNotFoundError: No module named 'harbor.models'`. In a dirty worktree, use `uv run --no-sync ...` after the editable install so machine-local uv settings cannot rewrite the repository's `uv.lock`. Do not export `UV_FROZEN` for an eval: subprocess runtimes inherit it, and their unlocked inline uv scripts cannot start in frozen mode. Run an eval from the repository root, not from inside its virtualenv. NLTK's import protection rejects dependencies installed beneath the current working directory. For a separate temporary venv, activate it by absolute path and run `uv run --active --no-project eval ...` from outside that venv. After dependency edits, sync the environment project before testing imports: ```bash uv sync --project path/to/env --all-extras ``` If a fresh environment import fails because resolved package versions are incompatible, fix the environment's `pyproject.toml` bounds and sync again. Do not patch generated `.venv` files. ## Smoke test Run a `3x1` (3 tasks, 1 rollout each) in plain-log mode to confirm the env loads and scores end-to-end. Spanning a few tasks surfaces weird things (bad rows, edge-case prompts, inconsistent scoring) that a single task hides: ```bash uv run --no-sync eval -n 3 -r 1 --no-rich -v --no-push ``` `--no-rich` turns off the live dashboard (which is on by default) for plain logs, `-v` prints prompts/completions, and `--no-push` keeps development runs local. This is the inner loop while developing — re-run it after each change to verify the dataset loads, the rollout runs, and the rubric scores as expected. Fix any errors here before scaling up. ## Full eval Run all tasks (omit `-n`) and pick `-r` so the total sample is representative — usually **>500 total rollouts** (`num_tasks × r`) is a good target. For large datasets `-r 1` may already clear that; for small ones, bump `-r`. Recommended full-eval invocation: ```bash uv run eval -r -c --no-rich ``` The run is **always saved** to disk (see [Inspect output](#inspect-output)) — there's no save flag. ### Key flags (`eval`) | Flag | Meaning | When | | ------ | --------- | ------ | | `-m ` | model id (default `deepseek/deepseek-v4-flash`) | to override the default | | `-n ` | number of tasks; **omit for all** | smoke (`-n 3`) vs full (omit) | | `-r ` | rollouts per task (>=2 if the taskset has `@group_reward`s) | almost always set | | `-c ` | max episodes in flight (default `128`) | raise for cheap envs, lower for sandboxed | | `-s` | **shuffle** tasks before taking the first `-n` | sampling a subset of a big dataset | | `-v` | debug logs (prompts/completions) | developing/debugging | | `--no-rich` | plain logs instead of the live dashboard | non-interactive or captured runs | | `-o ` | directory that groups runs; each run writes below its `run.dir` | pinning a known output root | | `--resume` | re-run a saved config's missing or errored episodes in place | use with `@ /configs/resolved/eval.json` | Most tasksets yield records as they are consumed. In Verifiers 0.3.1, `-n N` uses a lazy head before the runner collects those `N` tasks, but `automationbench_env`, `general_agent`, `tau2_bench`, and `tau3_bench` still construct their complete task lists. `-s` calls `Taskset.shuffle()` first, which must materialize the complete finite taskset, so avoid `-s` when a smoke test should construct only a prefix. ### Configuring the env Pass typed, dotted flags or a TOML file — there is no `-a` JSON blob: ```bash uv run eval wikispeedia --env.taskset.min-path-length 5 --env.taskset.max-path-length 8 uv run eval @ eval.toml # a launch TOML is re-runnable ``` Common knobs: `--env.agent.max-turns`, `--env.agent.max-total-tokens`, `--sampling.max-tokens`, `--sampling.temperature`, `--env.agent.harness.id`, `--env.agent.runtime.type`. `uv run eval -h` prints the full typed help, narrowed to the chosen taskset/harness. `TasksetConfig.system_prompt` is reserved for `--env.taskset.system-prompt`, a path to a prompt file that overrides every task during iteration. Environment configs must not redefine it as inline text; use a distinct field such as `task_system_prompt` for an inline/default task prompt. List-valued flags on union configs (e.g. `--env.agent.runtime.labels` on the prime runtime) must be passed as a space-separated JSON value — `--env.agent.runtime.labels '["ci"]'`. The bare form (`... labels ci`) and the `=`-form with JSON both parse as a string and fail validation with `Input should be a valid list`. A dotted override below a nested `RuntimeConfig` can replace that whole runtime with the selected provider's defaults. After overriding any nested runtime field, inspect `configs/resolved/eval.json` and confirm its image, workdir, runtime type, and resources before treating the smoke test as representative. ### Harness & runtime (sandboxed / agentic tasksets) A harness belongs to an agent seat: on the default single-agent env the seat is `agent` (`--env.agent.harness.*`); a multi-agent env names its own seats (e.g. `--env.solver.harness.*` and `--env.judge.harness.*` on `agentic-judge`). A taskset that bundles its own harness runs with it by default; otherwise pass `--env.agent.harness.id`. Select where rollouts execute with `--env.agent.runtime.type`: In Verifiers 0.3.1, the execution runtime belongs to the environment's agent role, not `HarnessConfig`. If a bundled harness must run with the interpreter that installed its package, export a custom `Env`/`EnvConfig` and set its agent default to `AgentConfig(runtime=SubprocessConfig())`. Toolsets running on Docker or Prime runtimes require Verifiers 0.3.1 to be installed from its source checkout, whether or not they are colocated with the harness. The framework builds and uploads its own source distribution with the environment package; a wheel-only Verifiers install has no `pyproject.toml` to build and fails before the tool server starts. Prime port exposure can return before its public route is reachable from another Prime sandbox. A taskset that supports a separate Prime tool-server runtime must check the exposed route from Prime before handing it to the agent; a localhost probe inside the server only proves that the process is listening. `TaskData.network_allow=[]` is the Verifiers 0.3.1 default-deny policy for Docker and Prime solver runtimes; Prime requires `vm=true`. Verifiers applies it after trusted task setup and retains its own interception and MCP routes. Subprocess and Modal do not implement task-data network policies, although a task-owned Modal runtime can set `network_access=false`. A secondary runtime created directly with `make_runtime()` must use a restricted config and call `prepare_execution(...)` after setup, before it runs agent-controlled code. ```bash uv run eval --env.agent.runtime.type subprocess # local process uv run eval --env.agent.runtime.type docker # local container (needs local docker) uv run eval --env.agent.runtime.type prime # remote prime VM (default; needs auth) uv run eval --env.agent.runtime.type modal # remote modal sandbox (needs auth) ``` Harbor tasksets should reuse `HarborTask` for environment upload and packaged `test.sh` scoring. When gated verifier inputs must not enter the agent box, export a `HarborEnv` subclass, set `HarborData.verifier`, and collect only the solution artifacts that the separate verifier needs. For packages whose task Dockerfile only selects a base image and copies task assets, resolve the `FROM` image, set `upload_environment=True`, and let `HarborTask.setup` stage the assets; Verifiers deliberately does not build task Dockerfiles. Harbor can also load an unpublished dataset directly from Git. Set `HarborConfig.dataset` to `""` and `repo` to a pinned GitHub URL of the form `https://github.com///tree//`; the implicit repository loader discovers the task directories, downloads them together, and caches the export. Reuse `HarborTaskset.load()` instead of implementing another archive downloader. RewardKit judges routed through an OpenAI-compatible endpoint need LiteLLM's provider prefix in front of the endpoint model id: a pinference model id such as `openai/gpt-5.5` is passed as `openai/openai/gpt-5.5`. Pass that model, the endpoint, and a credential template through `HarborData.verifier_env`; the packaged RewardKit verifier does not consume Verifiers judge sampling settings. Configure task-owned judge endpoints under `--env.taskset.task.judge.*`. Prime containers accept explicit registry refs such as GHCR, but Prime VM image builds require Docker Hub or Prime registry refs; point VM smoke tests at a published Prime mirror. The built-in `bash` harness (a bash + edit agent) is the fallback when no harness id is given; `null` is a tool-less chat loop (MCP tools only). For "bring your own search" tasksets that ship no search tool of their own (e.g. `openseeker`, `redsearcher`, `s1-deepresearch`, `arxivmath`), give the agent a Serper-backed `search` tool with `--env.agent.harness.search true` (needs `SERPER_API_KEY` in the eval environment): ```bash uv run eval openseeker --env.agent.harness.id bash --env.agent.harness.search true --env.agent.runtime.type prime ``` ### Concurrency `-c` caps episodes in flight (default `128`). An episode may contain multiple agent runs; those run one at a time by default. Set `--env.max-concurrent-agents ` only when an environment should play several agents from the same episode concurrently. The upper bound on simultaneous agent runs is therefore roughly `-c * --env.max-concurrent-agents`. Raise `-c` for cheap, non-sandboxed tasksets; keep it lower for sandboxed ones (containers/remote runtimes) and tune up from there. Eval uses the env-server pool by default; its settings live below `--serve.pool.*` (for example, `--serve.pool.type static --serve.pool.num-workers 4`), while `--serve.max-concurrent` optionally caps episodes per worker. Use `--no-serve` only when in-process execution is intentional. ### Models The taskset's **default model is usually fine** (`deepseek/deepseek-v4-flash`). Otherwise pick by tier (`-m `): - **Cheap** — `deepseek/deepseek-v4-flash` or `z-ai/glm-5.1`. - **Good (more capable)** — `openai/gpt-5.4` or `openai/gpt-5.5` at medium reasoning. - **Very hard only** — `openai/gpt-5.5` is the strongest but **expensive**. Avoid it by default; use it only as a last resort to confirm a task is solvable when the cheaper models get zero reward. ## Live dashboard in tmux The Rich dashboard is on by default and shows a live `reward · err ` headline, but it doesn't render in a plain captured shell. To watch it live, run inside tmux and drive it with `send-keys`: ```bash tmux new-session -d -s eval tmux send-keys -t eval 'uv run eval -r ' Enter # watch it: tmux attach -t eval # or: tmux capture-pane -t eval -p ``` For headless/automated runs, prefer `--no-rich` and follow the logs directly. ## Inspect output Each run is saved to a fresh per-run dir (so runs never overwrite each other): ```text outputs// configs/eval.toml # launch config when the run starts from CLI flags configs/resolved/eval.json # complete config; use this to resume traces.jsonl # one full episode per line logs/attempt_/eval.log # logs for one run or resume attempt logs/latest -> attempt_ # current attempt ``` ```bash # find the newest run ls -dt outputs/*/ | head # avg reward across rollouts (recomputed — aggregates aren't stored) jq -s '[.[].traces[] | [.rewards[]? | .score * .weight] | add // 0] | if length > 0 then add / length else 0 end' outputs/<...>/traces.jsonl ``` The dashboard shows the avg reward live; with `--no-rich`, recompute it from `traces.jsonl` (each line is an episode whose `traces[]` carry the per-reward breakdown). Skim a few episodes for sanity before publishing.