Operate Benchmark Lab

Operate the benchmark lab

The operator's manual for the whole benchmark/experiment lifecycle a coding agent can drive end to end. Two interfaces over the same sidecar files: the benchmarks MCP server (preferred for agents — same validation code as the hub API) and the CLI verbs. Execution always happens in a separate executor process; the MCP server and hub never run models. Command matrix, artifact map, and daemon details in reference.md; the tool table and agent loop in docs/agent-operator-surface.md.

Resolve CLI

Prefer the installed understudy binary. If it is unavailable inside a repo checkout, run through the package script:

npm run build
node dist/bin.js benchmarks mcp --root <dir>

MCP registration (Claude Code ~/.claude.json → mcpServers): { "understudy-benchmarks": { "type": "stdio", "command": "understudy", "args": ["benchmarks", "mcp"] } } — default root ~/.understudy/benchmarks; add --root per extra directory.

Safety Gates

• Queueing is not executing. queue_run / understudy runs queue only writes a request file. Model rollouts spend gateway money only when an executor picks the request up; say which executor will, before queueing.
• Trust posture, not per-call dialogs. Spend-adjacent shapes (multi-arm or multi-rollout runs, implicit all-task runs, experiment approval/verdict patches) consult the one-time posture in ~/.understudy/trust.json (understudy trust set, levels local_sandbox < bounded_experiments < hosted_ops). At bounded_experiments+ they proceed with a visible one-line notice (arm count, rough cost) — surface that notice to the user, then keep moving. Below that, the guard returns the one action to offer (understudy trust set bounded_experiments); confirm: true after explicit in-chat consent stays a per-call escape hatch. There is NO default spend cap: the posture's allow_spend_usd_per_run is an opt-in generous stop-loss (warn at 1x with a recorded spend_warning; hard stop only at 2x with spend_stop).
• Local arms are machine-aware. On predicted OOM (onboarding profile / memory probe) or a serve failure, the executor runs the arm on the gateway base model and records it — arm_fallback event plus fallback_reason on every row. Report the fallback; never present a fallen-back arm as a local measurement.
• One executor per benchmark dir. Before starting runs execute --watch, check for a live claim (claimed_by on the request, executor_version on events) — a stale watcher built before a feature landed is the classic corruption hazard; new requests carry requires:[...] so old executors skip them with run_unsupported instead of running them bare.
• Reviews and feedback are append-only ledgers; never edit reviews.jsonl/feedback.jsonl lines in place. Generated tasks are born accepted (review-policy default_decision: "accept") — reviews.jsonl carries explicit overrides only. apply_auto_accepts matters only for benchmarks opted into default_decision: "pending", and is itself the explicit user action — invoke it only when the developer asked.
• Honest reporting only: anomaly rows (rollout_timeout, app_replay_unobserved, structural sentinels) are excluded from aggregates but reported, never fabricated as scores. Overlapping CIs are a tie.
• Verifier-only changes regrade, never rerun. When only the verifier group changed (gold, contract, rubric, metric config — a MINOR bump), re-score the existing trajectories; queueing fresh rollouts for a verifier fix wastes gateway money and destroys comparability. Rerun is reserved for env-group (MAJOR) changes.
• Fixture-test every verifier before trusting it. One known-valid result must pass and one plausible-but-wrong result must fail. A verifier that has never rejected a wrong answer has not been tested; do not regrade or calibrate against it.

Outcome-first replacement loop

When the requested result is “meet or beat the incumbent” rather than merely “run this model,” operate one hash-bound decision loop:

1. Bind each task, history, trajectory, final effects, and outcome contract to one authoritative source execution. Do not optimize ambiguous or reconstructed rows.
2. Calibrate the verifier with a known-valid oracle and plausible wrong sentinels, then calibrate difficulty to prove measurable dev headroom.
3. Freeze immutable train, dev, and holdout boundaries. Optimize on train, make method decisions from canonical dev receipts, and leave holdout untouched until an explicitly authorized promotion evaluation.
4. Measure the incumbent and candidate through the same canonical serving contract. Preserve per-family scores, cost, latency, failures, and hashes; aggregate-only evidence is insufficient for regression gates.
5. Run GEPA only after those gates pass. Feed its canonical dev receipt into the fail-closed method ladder to decide whether to continue GEPA or advance to SFT, DPO, or GRPO within the remaining budget.
6. Before any promotion claim, require serving parity, strict arm evidence, protected-family no-regression, immutable artifacts, and an authorized fresh holdout receipt. target_met on dev is a candidate-selection result, not production evidence.

The exact contracts, failure boundaries, and current executor integration status are in references/outcome-first-replacement-loop.md. The method-ladder decision contract is in references/method-ladder.md.

The lifecycle

0. Intake (dropped workloads). When the user drops a folder or file into the chat (or names a local path), profile_workload scans it locally (payloads unread, nothing leaves the machine) and lists foundry-consumable dataset candidates. Discuss what the workload is, confirm the dataset and label/input columns with the user, then from_dataset compiles it into a PROPOSED benchmark (review-pending, executable: false) whose tasks land in the review inbox. Never queue a run from an intake — the user reviews the proposal first (step 2). 0.5. Interview before bulk authoring. Never batch-author tasks straight from an intake or trace corpus. First, from the trace clusters, propose 2–3 capability directions, each as a card: Name (the capability being measured), Example (one concrete task drawn from the traces), Tests (what the verifier would check), Needs (dependencies — services, fixtures, tools the environment must supply). The user picks a direction, then approves a posture per dependency: live (real service; usually wrong for a benchmark), frozen (recorded responses), or simulated (seeded synthetic implementation). Then build one task first: run it end to end and audit both trajectories — the agent's (did the environment answer sensibly, did the task make sense) and the verifier's (did scoring reflect what actually happened). Fixture-test the verifier (one valid pass, one plausible-wrong fail — see Safety Gates). Only after that pilot task survives audit do you batch-author the rest.
1. Build. Captured traces → understudy traces build-benchmark, then understudy traces author-tasks (gateway) for legible task definitions. The generated environment already ships the fixtures pre/post split (candidate-readable fixtures.json vs scorer-only gold.json) and the tiered gold-leakage audit (tier 1 verbatim findings, tier 2 fuzzy advisory) recorded in manifest.leakage_audit — read it, don't re-derive.
2. Review (born accepted). list_benchmarks → read_benchmark → read_task. Generated tasks are accepted by default — nothing to apply. read_benchmark returns each task's effective_decision plus attention_flags (low confidence, self-check, incumbent-failed, schema conflict, anomaly): work the flagged tasks, tweaking via submit_feedback or overriding via submit_review (reject | needs_more | restrict, or accept to re-accept an overridden task; newest line per task wins). apply_auto_accepts exists only for benchmarks running review-policy.json default_decision: "pending" (the old explicit-accept flow). Task or environment wrong? submit_feedback appends the ledger entry and returns the regenerate-env handoff (understudy traces regenerate-env).
3. Promote and calibrate. Gate promotion on understudy benchmarks rigor <dir> --ci (nonzero exit = findings to fix first), then understudy traces promote unlocks full runs. Queue the incumbent (--incumbent / incumbent_models) plus trivial_arms: ["null_agent", "spam_agent"]. Read calibration.json: null_floor/spam_floor with floor_exceeded: true (> 5%) names the passed_task_ids to fix; incumbent-failed tasks are flagged suspect — fix the benchmark before trusting any candidate score.
4. Run candidates and prompt experiments. queue_run with candidate models; prompt experiments ride prompt_overrides (--prompt-override <arm_label>=<model>=<suffix-file>, e.g. an SOP suffix appended to each task's system prompt) so prompt and model arms land in the same run under distinct arm labels.
5. Execute. Make sure exactly one executor is running: understudy runs execute --benchmark <dir> --watch (daemon; polls every 30s) or omit --watch for a single pass. Poll run_status until rows land; rollout_timeout kills hung rollouts into anomaly rows.
6. Read results rigorously. read_rollout / diff_rollouts (tool_sequence.diverges_at, unmet obligations), then understudy benchmarks rigor <dir> → rigor-report.md — the ABC attestation (oracle solvability, floors, calibration, contract complexity, anomalies, contamination provenance; honest UNKNOWN rows for unchecked items). Bootstrap CIs per docs/benchmark-rigor.md.
7. Regression after code edits. When the fix was to the user's app, run the app_replay arm via ../replay-app-harness/SKILL.md — same frozen tasks, current code, rows never feed calibration.

Versioning lifecycle (rerun / regrade / reuse)

Every task carries an optional semver version plus content_hashes (env_sha256, verifier_sha256, meta_sha256) — three canonical-JSON sha256 hashes over the field groups defined in docs/benchmark-rigor.md. The contract (mirrors Harbor's model):

Benchmark-level version is the max bump across tasks (added tasks = MAJOR, removed = MINOR). Every bump appends one understudy.benchmark_version.v1 line to versions.jsonl (append-only sidecar; records the bump, per-task reasons, splits hash, contamination status).

Operate it with three verbs:

• understudy runs regrade --benchmark <dir> [--run <id>] [--task <id>...] [--dry-run] — verifier-only change: re-scores retained trajectory evidence offline against the CURRENT verifier and writes fresh rows under <run>-regrade-<n> (source-run provenance, original cost/latency preserved), then appends one MINOR versions.jsonl line for the regraded tasks so the superseded source rows go stale instead of double-counting next to their regrades. Never queues rollouts, never spends gateway money. If you're about to rerun after a gold/rubric fix, stop — regrade instead.
• understudy benchmarks upgrade <dir> --against <old-benchmark.json> — diffs the current manifest against the archived previous one and prints the minimal plan first: which task ids rerun, which regrade, which reuse, and the resulting version bump; then appends one versions.jsonl line (--dry-run skips the append). Nothing executes; --queue --model <id> writes run requests for the rerun set only.
• understudy benchmarks rigor <dir>... --ci — the gate before understudy traces promote and before publishing any new version: exits nonzero on hard findings (floors exceeded, oracle unsolved, verbatim leakage, contamination) instead of just writing the report; UNKNOWN is fatal only with --strict. Run it in the upgrade path too — a version bump that fails rigor does not promote (traces promote enforces this; --override-rigor <reason> is recorded in the promotion record).

Leaderboard staleness. A row is stale when it predates the newest breaking bump for its task (MAJOR or MINOR line in versions.jsonl; rows without created_at are conservatively stale). MAJOR means stale — rerun (the trajectory itself is invalid); MINOR means stale — regrade (trajectory reusable, score not); PATCH-only changes leave rows current. Report stale rows the honest-reporting way: excluded from headline aggregates, counted and named, never silently dropped or silently included.

Daemon lifecycle (executor + desktop)

• Check first: ps/events for a live runs execute watcher; the first stderr line prints executor version <v> (pid <n>). A live foreign claim is respected; a dead claim is taken over (staleness takeover).
• Start: understudy runs execute --benchmark <dir> --watch in the background; single-pass (no --watch) for one drain. Stop safely by killing the watcher between polls — requests are re-claimable; in-flight rollouts past the timeout become anomaly rows, never silent hangs.
• Desktop app daemon (separate thing — local model serving, not the run queue): understudy daemon status pid-checks and health-probes ~/.understudy/agent-card.json; then understudy desktop capabilities / status / model|slot|download|chat verbs. See reference.md for what the app covers vs headless CLI.

Output Standard

End with: benchmark dir and stage (proposed/promoted); current benchmark version (semver, from the manifest / last versions.jsonl line) and stale-row status (rows counted by current / stale-regrade / stale-rerun, and whether any stale rows sit in a headline aggregate); ledger state (reviews, auto-accepts, feedback); calibration verdict (incumbent gate, null/spam floors, suspects); runs queued/executed and by which executor version; result type (validation, oracle, live, app-replay); artifact paths written (calibration.json, rigor-report.md, rows-*.jsonl, versions.jsonl); and one recommended next command.