Compare Tasks

Compare two harbor runs on one task

Use when given two harbor run names and a task name, and the goal is to understand why the two runs differ on that task — not just that they differ.

Inputs

• RUN_A: harbor run name (e.g. sonnet46-full)
• RUN_B: harbor run name (e.g. pi-sonnet46-full)
• TASK: bare task name (e.g. extract-elf, not terminal-bench/extract-elf)
• RUNS_DIR: defaults to evals/harbor/runs/ relative to the repo root

Procedure

1. Find each run's trial directory for the task

Harbor 0.8 names trial dirs <task>__<random-suffix> (e.g. extract-elf__bU3GHs4), not <task>.1. The suffix is unique per trial, so don't guess it — discover it from disk:

TRIAL_A_DIR=$(ls -d "$RUNS_DIR/$RUN_A/${TASK}__"*/ 2>/dev/null | head -1)
TRIAL_B_DIR=$(ls -d "$RUNS_DIR/$RUN_B/${TASK}__"*/ 2>/dev/null | head -1)

If either is empty, that run didn't include this task — stop and say so. (ls "$RUNS_DIR/$RUN_A/" shows what's there.)

If you want to confirm the match, every result.json carries task_name and trial_name:

jq '{task_name, trial_name}' "$TRIAL_A_DIR/result.json"

2. Headline facts

The fastest path is to let cmd.py task do it for you — it already prints status, reward, duration, tokens, turns, cost, error class, and the tail of the verifier stdout:

./evals/harbor/cmd.py task "$RUN_A" "$TASK"
./evals/harbor/cmd.py task "$RUN_B" "$TASK"

Only drop to raw jq against result.json if you need a field cmd.py task doesn't print. The actual shape (harbor 0.8 TrialResult):

jq '{
  reward: (.verifier_result.rewards.reward
           // (.verifier_result.rewards | to_entries | .[0].value)
           // null),
  rewards_all: .verifier_result.rewards,
  duration_seconds: ((.finished_at | fromdateiso8601) - (.started_at | fromdateiso8601)),
  input_tokens: .agent_result.n_input_tokens,
  cache_tokens: .agent_result.n_cache_tokens,
  output_tokens: .agent_result.n_output_tokens,
  cost_usd: .agent_result.cost_usd,
  error_type: .exception_info.exception_type,
  error_message: (.exception_info.exception_message // "" | split("\n")[0])
}' "$TRIAL_A_DIR/result.json"

The reward fallback mirrors reporter.trial_reward: if the verifier didn't use the conventional reward key, take the first value in the rewards map.

Derive status from those:

• pass if reward >= 1.0
• partial if reward > 0 (and < 1)
• fail if reward == 0
• timeout if reward is 0/null and error_type contains "timeout"
• error if reward is 0/null and error_type is set (non-timeout)
• no-reward if neither verifier_result.rewards nor exception_info is set

Reward wins over errors: harbor can record an AgentTimeoutError after the verifier already scored a pass (the agent finished the work then the harness timed out during teardown, or it timed out after writing the correct answer). If we got points, count them. See reporter.trial_status for the canonical rule.

Several agent_result fields can be null depending on the harness (notably n_cache_tokens, n_output_tokens, cost_usd on some goose runs). Don't treat that as a failure — just omit those facts from the comparison if missing on either side. cmd.py task already applies harbor's fallbacks (reading goose's complete event from agent/goose.txt when the structured field is null), so its numbers are the right ones to report.

3. Read the task spec

The task definitions are NOT in the harbor Python package. They are plain text files on disk, in harbor's task cache. Do not run find / or pip show harbor — that is the wrong direction.

Harbor caches under ~/.cache/harbor/ on every platform (it uses Path("~/.cache/harbor").expanduser() unconditionally — there is no ~/Library/Caches/harbor on macOS, despite what you might expect).

The on-disk layout for package-backed tasks (the common case — everything in terminal-bench/terminal-bench-2 lands here) is:

~/.cache/harbor/tasks/packages/<org>/<task>/<digest>/

Note: no dataset name in the path. Tasks are keyed by org + task name + content digest, not by which dataset pulled them. The <digest> segment changes when the task is republished, so discover the dir rather than hardcoding:

TASK_DIR=$(ls -d ~/.cache/harbor/tasks/packages/terminal-bench/"$TASK"/*/ 2>/dev/null | head -1)
echo "$TASK_DIR"
ls "$TASK_DIR"

If that's empty, the task could be from a different org or a git source — broaden the search. find returns the parent (one level above the digest), so descend one more level. Guard against $PARENT being empty, otherwise the glob expands to /*/ and matches the filesystem root:

PARENT=$(find ~/.cache/harbor/tasks -type d -name "$TASK" 2>/dev/null | head -1)
if [ -n "$PARENT" ]; then
  TASK_DIR=$(ls -d "$PARENT"/*/ 2>/dev/null | head -1)
fi

If both lookups come up empty, the task hasn't been downloaded on this machine — bail out and report that, rather than guessing. (Runs sync via cmd.py pull but the task cache does not, so a machine that only inspects results may never have the spec locally.)

~/.cache/harbor/datasets/ exists too but holds dataset-level metadata, not the per-task instruction.md / tests/ / solution/ files — not what you want here.

Inside, you care about three files:

• instruction.md — exactly what the agent was asked to do
• tests/test_outputs.py (or sometimes run-tests.sh) — what the verifier actually checks, line by line
• solution/solution.sh — the reference correct answer

Without all three you can't tell whether a wrong answer was a misread, a shallow bug, or a verifier surprise. Quote the assertion that failed when you describe a failure — paraphrasing is how wrong conclusions sneak in.

4. Read each agent's trajectory

Two sources, prefer the first when present:

• $TRIAL_DIR/agent/trajectory.json — harbor's ATIF format, one entry per agent step. jq '.steps[] | {step_id, source, message, tool_calls: [.tool_calls[]?.function_name]}' gives a compact view. Most current runs have it; some older harness versions may not.
• $TRIAL_DIR/agent/<harness>.txt — raw stream-json or log. The filename matches the harness (commonly goose.txt or pi.txt; other harnesses use their own name). Don't guess — run ls "$TRIAL_DIR/agent/" and use whatever .txt file is there.

Skim, don't quote in full. For each agent identify:

• the approach it took (e.g. "wrote a Python script that walks the ELF section headers")
• the final artifacts it left in the container (file paths it created or modified)
• for losers, the failure mode — one of:
  • misread the spec (wrong assumption about input/output)
  • right approach, shallow bug (off-by-one, wrong encoding, wrong base address)
  • ran out of clock (timeout) — note whether it was still making progress or had gone in circles
  • diverged into an unproductive thread (e.g. debugging a non-issue)
  • the verifier expected something the spec didn't telegraph

5. Read the verifier output

$TRIAL_DIR/verifier/ typically contains:

• test-stdout.txt — the verifier's full stdout (assertion failures, pytest output, etc.). This is usually the most diagnostic file.
• reward.txt — the scalar reward as a string.
• ctrf.json — structured test results in CTRF format, useful if you want per-assertion pass/fail without grepping stdout.

tail -50 "$TRIAL_DIR/verifier/test-stdout.txt"

This is often more diagnostic than the agent log — it tells you exactly which assertion failed and what the agent's output was at that point.

6. Produce the comparison

Output markdown with these sections in order:

• Headline (1 line): who won, by how much (reward + cost / duration if meaningful, omitting fields that are null on either side).
• What A did (2-4 sentences): plan, final artifact, verifier outcome.
• What B did (2-4 sentences): same shape as A.
• Why outcomes differ (2-4 sentences): the actual mechanism. Not "B was smarter" but "B's script used nm -n so its addresses matched the verifier's ground truth, A's script used PIE-relocated virtual addresses which the verifier doesn't normalize".
• Generalizable lesson (optional, 1-2 sentences): is this a pattern that probably affects other tasks, or a one-off accident of this verifier? Skip if unclear from one task.

Tools you'll need

• ./evals/harbor/cmd.py task <run> <task> for the headline numbers
• ls -d to discover the <task>__<suffix> trial directories
• jq for any result.json field cmd.py task doesn't print
• file reads against $TRIAL_DIR/agent/ and $TRIAL_DIR/verifier/
• find ~/.cache/harbor/tasks to locate the task spec

No Python imports, no harbor package required. Everything you need is on disk as JSON / text files.

Source: aaif-goose/goose — distributed by TomeVault.