Run Eval

Run a single eval

The easy entrypoint for one eval run: one Task × one Model × one AgentConfig. A single run is just a 1×1×1 matrix, driven through the same wrapper everything else uses. This skill keeps the front-end lean — no concurrency, no combo math, no cross-combo safety. For those, use a different skill (see Wrong tool?).

The run mechanics are shared and live in the references — read them, don't re-derive them here:

• Local vs bastion, auth, clean pre-flight, launch, knobs, results layout → ../../references/running-evals.md
• Capability → tool mapping for your harness → ../../references/harness-capabilities.md

Flow

1. Choose where + how, then pin the one combo

• Local or bastion? Local is the default; remote sets BENCH_REMOTE=1 + the BASTION_* connection env. Auth: ambient cloud credentials (BENCH_VERTEX=1, no keys) or API keys — pick one. Both covered in running-evals.md.
• Pin exactly one Task (task.yaml path), one Model, and one AgentConfig (oc|gcli [+mcp][+skills]; oc = OpenClaw, gcli = Gemini CLI), driven through scripts/bastion/run_matrix.sh.
• Ask the operator for anything not given — don't guess on the dimensions that cost a cluster + time (budget roughly half an hour for an infra-bearing task).

2. Preview with DRY_RUN=1

Show the expanded (single) combo + per-combo env before spending a cluster:

DRY_RUN=1 \
MATRIX_TASKS="tasks/common/opa-remediation/task.yaml" \
MATRIX_MODELS="gemini-3.1-pro-preview" \
MATRIX_AGENT_CONFIGS="gcli+mcp+skills" \
  scripts/bastion/run_matrix.sh

3. Clean pre-flight

Work the "Before any retry" checklist in ../../../docs/appendix/known_issues.md before launching — stale per-run state and orphaned cloud resources are the top cause of a "fresh" run failing instantly. Keep it scoped to your run; for leaked clusters / node service accounts (e.g. gke-nodes-*) / secrets, use the cleanup-orphaned-resources skill.

4. Launch the one combo (detached)

Drive the wrapper with single-value MATRIX_*. It runs detached under nohup and prints a STAMP — record RESUME_STAMP=<stamp> in durable state; it is your handle for monitoring and re-attach. Example (API keys from ~/secrets.env, local runner; for ambient credentials add BENCH_VERTEX=1, and prefix BENCH_REMOTE=1 + BASTION_* for remote — see running-evals.md for both variants):

PROJECT_ID=<proj> \
MATRIX_TASKS="tasks/common/opa-remediation/task.yaml" \
MATRIX_MODELS="<model-id>" \
MATRIX_AGENT_CONFIGS="oc+mcp+skills" \
RESULTS_DIR="results/<label>" \
  scripts/bastion/run_matrix.sh

(MAX_PARALLEL is irrelevant for one combo.) Full launch details, including the matrix knobs and the judge/auth settings, are in running-evals.md.

5. Watch it

Basic loop: poll the combo's status + run.log tail on an interval (~3–5 min for an infra-bearing task — don't busy-poll). If your poller dies the detached run continues; re-attach with RESUME_STAMP=<stamp> and the same command. For a resilient, hands-off watch that survives quiet stretches and dead watchers, follow ../../references/monitoring-and-recovery.md (for one combo it degenerates to one watcher + the keepalive loop).

Classify a flake vs a real failure against the router in known_issues.md: infra flake → clean + retry (cap 2) — a retry is a new launch, so run it without RESUME_STAMP (set, the wrapper attaches to the failed run instead of launching) and record the new STAMP as the active attempt; real failure (auth/config, low score, task-logic) → do not retry, analyze.

6. Summarize + where to read scores

When the combo has a terminal status (or .done), report: task · model · agent-config · auth-mode · exit · score · pass/fail checks. Results land per the layout in running-evals.md; for how scoring works and how to interpret it, see ../../../docs/components/metrics.md. If it didn't pass, give the decisive log line (redact secrets), the root cause, and the concrete fix — the diagnose-eval-failure skill covers the "why did the model score low" half. Then confirm teardown is clean.

Wrong tool?

• More than one combo (a matrix, a comparison) → use the run-parallel-evals skill — it adds combo math, MAX_PARALLEL, and cross-combo parallel-safety.
• Explaining a low score on a graded run → the diagnose-eval-failure skill.
• Cleaning up after an aborted run → the cleanup-orphaned-resources skill.

Guardrails

• One run still costs a real cluster (cloud, or kind on the runner host) plus tens of minutes — confirm scale and DRY_RUN=1 first.
• Never print or commit API keys; redact secrets in summaries.
• Never launch a second run of the same task+model+config concurrently — the run-id-derived cluster name is deterministic in the combo, so both runs would target the same cluster.
• Always confirm clean teardown — a leftover cluster or node service account (e.g. gke-nodes-*) from a failed teardown makes a retry of the same combo fail with 409 already exists.
• Prefer leaving the run detached + re-attaching via RESUME_STAMP over a fragile foreground session.