# Operate Benchmark Lab

> Use when a coding agent must operate the full benchmark lifecycle over local benchmark dirs — "build a benchmark from my traces and run models on it", "review and calibrate the eval", "queue a prompt experiment", "is an executor running", "read the rigor report". Covers traces → build-benchmark → review/feedback → calibration floors → candidate and prompt-override runs → rigor/CI reading → app-replay regression, via the benchmarks MCP server or CLI verbs, plus the run-executor daemon lifecycle.

- Skill: `understudylabs/operate-benchmark-lab` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add understudylabs/operate-benchmark-lab`
- Raw SKILL.md: https://api.skillmd.com/api/skills/understudylabs/operate-benchmark-lab/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: understudylabs (https://skillmd.com/u/understudylabs)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/understudylabs/operate-benchmark-lab

---


# Operate the benchmark lab

The operator's manual for the whole benchmark/experiment lifecycle a coding
agent can drive end to end. Two interfaces over the same sidecar files:
the **benchmarks MCP server** (preferred for agents — same validation code as
the hub API) and the **CLI verbs**. Execution always happens in a separate
executor process; the MCP server and hub never run models. Command matrix,
artifact map, and daemon details in [`reference.md`](reference.md); the tool
table and agent loop in `docs/agent-operator-surface.md`.

## Resolve CLI

Prefer the installed `understudy` binary. If it is unavailable inside a repo
checkout, run through the package script:

```sh
npm run build
node dist/bin.js benchmarks mcp --root <dir>
```

MCP registration (Claude Code `~/.claude.json` → `mcpServers`):
`{ "understudy-benchmarks": { "type": "stdio", "command": "understudy", "args": ["benchmarks", "mcp"] } }`
— default root `~/.understudy/benchmarks`; add `--root` per extra directory.

## Safety Gates

- **Queueing is not executing.** `queue_run` / `understudy runs queue` only
  writes a request file. Model rollouts spend gateway money only when an
  executor picks the request up; say which executor will, before queueing.
- **Trust posture, not per-call dialogs.** Spend-adjacent shapes (multi-arm
  or multi-rollout runs, implicit all-task runs, experiment
  approval/verdict patches) consult the one-time posture in
  `~/.understudy/trust.json` (`understudy trust set`, levels `local_sandbox`
  < `bounded_experiments` < `hosted_ops`). At `bounded_experiments`+ they
  proceed with a visible one-line notice (arm count, rough cost) — surface
  that notice to the user, then keep moving. Below that, the guard returns
  the one action to offer (`understudy trust set bounded_experiments`);
  `confirm: true` after explicit in-chat consent stays a per-call escape
  hatch. There is NO default spend cap: the posture's
  `allow_spend_usd_per_run` is an opt-in generous stop-loss (warn at 1x with
  a recorded `spend_warning`; hard stop only at 2x with `spend_stop`).
- **Local arms are machine-aware.** On predicted OOM (onboarding profile /
  memory probe) or a serve failure, the executor runs the arm on the gateway
  base model and records it — `arm_fallback` event plus `fallback_reason` on
  every row. Report the fallback; never present a fallen-back arm as a local
  measurement.
- **One executor per benchmark dir.** Before starting `runs execute --watch`,
  check for a live claim (`claimed_by` on the request, `executor_version` on
  events) — a stale watcher built before a feature landed is the classic
  corruption hazard; new requests carry `requires:[...]` so old executors skip
  them with `run_unsupported` instead of running them bare.
- Reviews and feedback are append-only ledgers; never edit
  `reviews.jsonl`/`feedback.jsonl` lines in place. Generated tasks are born
  accepted (review-policy `default_decision: "accept"`) — `reviews.jsonl`
  carries explicit overrides only. `apply_auto_accepts` matters only for
  benchmarks opted into `default_decision: "pending"`, and is itself the
  explicit user action — invoke it only when the developer asked.
- Honest reporting only: anomaly rows (`rollout_timeout`,
  `app_replay_unobserved`, structural sentinels) are excluded from aggregates
  but reported, never fabricated as scores. Overlapping CIs are a tie.
- **Verifier-only changes regrade, never rerun.** When only the verifier
  group changed (gold, contract, rubric, metric config — a MINOR bump),
  re-score the existing trajectories; queueing fresh rollouts for a verifier
  fix wastes gateway money and destroys comparability. Rerun is reserved for
  env-group (MAJOR) changes.
- **Fixture-test every verifier before trusting it.** One known-valid result
  must pass and one plausible-but-wrong result must fail. A verifier that has
  never rejected a wrong answer has not been tested; do not regrade or
  calibrate against it.

## Outcome-first replacement loop

When the requested result is “meet or beat the incumbent” rather than merely
“run this model,” operate one hash-bound decision loop:

1. Bind each task, history, trajectory, final effects, and outcome contract to
   one authoritative source execution. Do not optimize ambiguous or
   reconstructed rows.
2. Calibrate the verifier with a known-valid oracle and plausible wrong
   sentinels, then calibrate difficulty to prove measurable dev headroom.
3. Freeze immutable train, dev, and holdout boundaries. Optimize on train,
   make method decisions from canonical dev receipts, and leave holdout
   untouched until an explicitly authorized promotion evaluation.
4. Measure the incumbent and candidate through the same canonical serving
   contract. Preserve per-family scores, cost, latency, failures, and hashes;
   aggregate-only evidence is insufficient for regression gates.
5. Run GEPA only after those gates pass. Feed its canonical dev receipt into
   the fail-closed method ladder to decide whether to continue GEPA or advance
   to SFT, DPO, or GRPO within the remaining budget.
6. Before any promotion claim, require serving parity, strict arm evidence,
   protected-family no-regression, immutable artifacts, and an authorized
   fresh holdout receipt. `target_met` on dev is a candidate-selection result,
   not production evidence.

The exact contracts, failure boundaries, and current executor integration
status are in [`references/outcome-first-replacement-loop.md`](references/outcome-first-replacement-loop.md).
The method-ladder decision contract is in
[`references/method-ladder.md`](references/method-ladder.md).

## The lifecycle

0. **Intake (dropped workloads).** When the user drops a folder or file into
   the chat (or names a local path), `profile_workload` scans it locally
   (payloads unread, nothing leaves the machine) and lists
   foundry-consumable dataset candidates. Discuss what the workload is,
   confirm the dataset and label/input columns with the user, then
   `from_dataset` compiles it into a PROPOSED benchmark (review-pending,
   `executable: false`) whose tasks land in the review inbox. Never queue a
   run from an intake — the user reviews the proposal first (step 2).
0.5. **Interview before bulk authoring.** Never batch-author tasks straight
   from an intake or trace corpus. First, from the trace clusters, propose
   **2–3 capability directions**, each as a card: **Name** (the capability
   being measured), **Example** (one concrete task drawn from the traces),
   **Tests** (what the verifier would check), **Needs** (dependencies —
   services, fixtures, tools the environment must supply). The user picks a
   direction, then approves a posture **per dependency**: `live` (real
   service; usually wrong for a benchmark), `frozen` (recorded responses), or
   `simulated` (seeded synthetic implementation). Then build **one task
   first**: run it end to end and audit *both* trajectories — the agent's
   (did the environment answer sensibly, did the task make sense) and the
   verifier's (did scoring reflect what actually happened). Fixture-test the
   verifier (one valid pass, one plausible-wrong fail — see Safety Gates).
   Only after that pilot task survives audit do you batch-author the rest.
1. **Build.** Captured traces → `understudy traces build-benchmark`, then
   `understudy traces author-tasks` (gateway) for legible task definitions.
   The generated environment already ships the fixtures pre/post split
   (candidate-readable `fixtures.json` vs scorer-only `gold.json`) and the
   tiered gold-leakage audit (tier 1 verbatim findings, tier 2 fuzzy
   advisory) recorded in `manifest.leakage_audit` — read it, don't re-derive.
2. **Review (born accepted).** `list_benchmarks` → `read_benchmark` →
   `read_task`. Generated tasks are accepted by default — nothing to apply.
   `read_benchmark` returns each task's `effective_decision` plus
   `attention_flags` (low confidence, self-check, incumbent-failed, schema
   conflict, anomaly): work the flagged tasks, tweaking via `submit_feedback`
   or overriding via `submit_review`
   (`reject | needs_more | restrict`, or `accept` to re-accept an overridden
   task; newest line per task wins). `apply_auto_accepts` exists only for
   benchmarks running `review-policy.json` `default_decision: "pending"`
   (the old explicit-accept flow).
   Task or environment wrong? `submit_feedback` appends the ledger entry and
   returns the regenerate-env handoff (`understudy traces regenerate-env`).
3. **Promote and calibrate.** Gate promotion on
   `understudy benchmarks rigor <dir> --ci` (nonzero exit = findings to fix
   first), then `understudy traces promote` unlocks full runs.
   Queue the incumbent (`--incumbent` / `incumbent_models`) plus
   `trivial_arms: ["null_agent", "spam_agent"]`. Read `calibration.json`:
   `null_floor`/`spam_floor` with `floor_exceeded: true` (> 5%) names the
   `passed_task_ids` to fix; incumbent-failed tasks are flagged suspect —
   fix the benchmark before trusting any candidate score.
4. **Run candidates and prompt experiments.** `queue_run` with candidate
   models; prompt experiments ride `prompt_overrides`
   (`--prompt-override <arm_label>=<model>=<suffix-file>`, e.g. an SOP
   suffix appended to each task's system prompt) so prompt and model arms
   land in the same run under distinct arm labels.
5. **Execute.** Make sure exactly one executor is running:
   `understudy runs execute --benchmark <dir> --watch` (daemon; polls every
   30s) or omit `--watch` for a single pass. Poll `run_status` until rows
   land; `rollout_timeout` kills hung rollouts into anomaly rows.
6. **Read results rigorously.** `read_rollout` / `diff_rollouts`
   (`tool_sequence.diverges_at`, unmet obligations), then
   `understudy benchmarks rigor <dir>` → `rigor-report.md` — the ABC
   attestation (oracle solvability, floors, calibration, contract
   complexity, anomalies, contamination provenance; honest UNKNOWN rows for
   unchecked items). Bootstrap CIs per `docs/benchmark-rigor.md`.
7. **Regression after code edits.** When the fix was to the user's app, run
   the `app_replay` arm via [`../replay-app-harness/SKILL.md`](../replay-app-harness/SKILL.md)
   — same frozen tasks, current code, rows never feed calibration.

## Versioning lifecycle (rerun / regrade / reuse)

Every task carries an optional semver `version` plus `content_hashes`
(`env_sha256`, `verifier_sha256`, `meta_sha256`) — three canonical-JSON
sha256 hashes over the field groups defined in `docs/benchmark-rigor.md`.
The contract (mirrors Harbor's model):

| Changed group | Bump | Consequence |
| --- | --- | --- |
| env (instruction, fixtures, tool surface, seed, unknown fields) | MAJOR | **rerun** — old trajectories are invalid |
| verifier (gold refs, contract/rubric, metric config) | MINOR | **regrade** — trajectories stand, re-score them |
| meta (title, docs, tags) | PATCH | **reuse** — existing rows as-is |

Benchmark-level `version` is the max bump across tasks (added tasks = MAJOR,
removed = MINOR). Every bump appends one
`understudy.benchmark_version.v1` line to `versions.jsonl` (append-only
sidecar; records the bump, per-task reasons, splits hash, contamination
status).

Operate it with three verbs:

- **`understudy runs regrade --benchmark <dir> [--run <id>] [--task <id>...]
  [--dry-run]`** — verifier-only change: re-scores retained trajectory
  evidence offline against the CURRENT verifier and writes fresh rows under
  `<run>-regrade-<n>` (source-run provenance, original cost/latency
  preserved), then appends one MINOR `versions.jsonl` line for the regraded
  tasks so the superseded source rows go stale instead of double-counting
  next to their regrades. Never queues rollouts, never spends gateway
  money. If you're about to rerun after a gold/rubric fix, stop — regrade
  instead.
- **`understudy benchmarks upgrade <dir> --against <old-benchmark.json>`** —
  diffs the current manifest against the archived previous one and prints
  the minimal plan first: which task ids **rerun**, which **regrade**, which
  **reuse**, and the resulting version bump; then appends one
  `versions.jsonl` line (`--dry-run` skips the append). Nothing executes;
  `--queue --model <id>` writes run requests for the rerun set only.
- **`understudy benchmarks rigor <dir>... --ci`** — the gate before
  `understudy traces promote` and before publishing any new version: exits
  nonzero on hard findings (floors exceeded, oracle unsolved, verbatim
  leakage, contamination) instead of just writing the report; UNKNOWN is
  fatal only with `--strict`. Run it in the upgrade path too — a version
  bump that fails rigor does not promote (`traces promote` enforces this;
  `--override-rigor <reason>` is recorded in the promotion record).

**Leaderboard staleness.** A row is stale when it predates the newest
breaking bump for its task (MAJOR or MINOR line in `versions.jsonl`; rows
without `created_at` are conservatively stale). MAJOR means **stale —
rerun** (the trajectory itself is invalid); MINOR means **stale — regrade**
(trajectory reusable, score not); PATCH-only changes leave rows current.
Report stale rows the honest-reporting way: excluded from headline
aggregates, counted and named, never silently dropped or silently included.

## Daemon lifecycle (executor + desktop)

- **Check first**: `ps`/events for a live `runs execute` watcher; the first
  stderr line prints `executor version <v> (pid <n>)`. A live foreign claim
  is respected; a dead claim is taken over (staleness takeover).
- **Start**: `understudy runs execute --benchmark <dir> --watch` in the
  background; single-pass (no `--watch`) for one drain. Stop safely by
  killing the watcher between polls — requests are re-claimable; in-flight
  rollouts past the timeout become anomaly rows, never silent hangs.
- **Desktop app daemon** (separate thing — local model serving, not the run
  queue): `understudy daemon status` pid-checks and health-probes
  `~/.understudy/agent-card.json`; then `understudy desktop capabilities` /
  `status` / `model|slot|download|chat` verbs. See
  [`reference.md`](reference.md) for what the app covers vs headless CLI.

## Output Standard

End with: benchmark dir and stage (proposed/promoted); **current benchmark
version** (semver, from the manifest / last `versions.jsonl` line) and
**stale-row status** (rows counted by current / stale-regrade / stale-rerun,
and whether any stale rows sit in a headline aggregate); ledger state
(reviews, auto-accepts, feedback); calibration verdict (incumbent gate,
null/spam floors, suspects); runs queued/executed and by which executor
version; result type (validation, oracle, live, app-replay); artifact paths
written (`calibration.json`, `rigor-report.md`, `rows-*.jsonl`,
`versions.jsonl`); and one recommended next command.