Manage Evals

Managing Evaluations

Overview

OpenHands evaluations produce results stored on a CDN at https://results.eval.all-hands.dev/. Each run is identified by a path: {benchmark}/{model_slug}/{github_run_id}/. This skill enables triggering evaluation runs, comparing results between runs, and posting performance reports as GitHub PR comments.

Quick Start

Trigger an Evaluation

python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
    --sdk-ref <BRANCH_OR_TAG> --benchmark swebench --eval-limit 50

Compare Runs

python .agents/skills/manage-evals/scripts/manage_evals.py compare \
    "<benchmark>/<model_slug>/<run_id>/" \
    --auto-baseline

Compare and Post to PR

python .agents/skills/manage-evals/scripts/manage_evals.py compare \
    "<benchmark>/<model_slug>/<run_id>/" \
    --auto-baseline \
    --post-comment --pr <PR_NUMBER> --repo OpenHands/software-agent-sdk

Triggering Evaluations

Using the Script

# SWE-bench (default) on a PR branch
python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
    --sdk-ref my-feature-branch --eval-limit 50

# GAIA benchmark
python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
    --sdk-ref main --benchmark gaia --eval-limit 50

# With a specific model
python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
    --sdk-ref v1.16.0 --benchmark swebench --model-ids gemini-3-flash --eval-limit 50

# Multiple benchmarks (run the command multiple times)
for bench in swebench gaia; do
    python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
        --sdk-ref main --benchmark "$bench" --eval-limit 50 --reason "Multi-benchmark eval"
done

Available Benchmarks

Trigger Options

Via PR Labels (Alternative)

Adding a label to a PR also triggers evaluations:

• run-eval-1 — 1 instance (quick sanity check)
• run-eval-50 — 50 instances (standard comparison)
• run-eval-200 — 200 instances
• run-eval-500 — 500 instances (full benchmark)

Comparing Evaluation Runs

Step 1: Find the Current PR's Eval Run

Eval runs are triggered by adding labels like run-eval-50 to a PR. The all-hands-bot posts a comment with results when complete.

Option A — From bot comments on the PR:

gh api repos/OpenHands/software-agent-sdk/issues/<PR_NUMBER>/comments \
    --jq '.[] | select(.user.login == "all-hands-bot") | .body' \
    | grep -o 'Evaluation:.*' | head -1

The evaluation name follows the format {github_run_id}-{model_slug_short} (e.g., 23775164157-claude-son). Extract the github_run_id from this.

Option B — From the "Evaluation Triggered" bot comment:

gh api repos/OpenHands/software-agent-sdk/issues/<PR_NUMBER>/comments \
    --jq '.[] | select(.body | test("Evaluation Triggered")) | .body'

This contains the SDK commit SHA. Cross-reference with daily metadata to find the run ID.

Option C — From daily metadata:

curl -s "https://results.eval.all-hands.dev/metadata/$(date -u +%Y-%m-%d).txt"

Each line is a run path. Match by benchmark and model to find the run.

Step 2: Identify the Run Path Components

A run path has three components:

• benchmark: swebench, swebenchpro, gaia, swtbench, commit0, swebenchmultimodal, terminalbench, programbench
• model_slug: Derived from model name with /:@. replaced by - (e.g., litellm_proxy-claude-sonnet-4-5-20250929)
• run_id: The GitHub Actions workflow run ID from the OpenHands/evaluation repo

Step 3: Verify Results Exist

curl -sI "https://results.eval.all-hands.dev/<benchmark>/<model_slug>/<run_id>/output.report.json" | head -1

A 200 status confirms the run completed and results are available.

Step 4: Find a Baseline for Comparison

Automatic: The comparison script's --auto-baseline flag scans metadata files backward up to 14 days to find the most recent completed run with the same benchmark and model.

Manual: Inspect metadata files or other PR bot comments to identify a specific run:

# Check today's runs
curl -s "https://results.eval.all-hands.dev/metadata/$(date -u +%Y-%m-%d).txt" | grep "swebench/litellm_proxy-claude"

# Check yesterday's runs
curl -s "https://results.eval.all-hands.dev/metadata/$(date -u -d yesterday +%Y-%m-%d).txt" | grep "swebench/litellm_proxy-claude"

Step 5: Run the Comparison

python .agents/skills/manage-evals/scripts/manage_evals.py compare \
    "swebench/litellm_proxy-claude-sonnet-4-5-20250929/23775164157/" \
    --baseline "swebench/litellm_proxy-claude-sonnet-4-5-20250929/23773892085/"

Or with auto-baseline and PR comment posting:

python .agents/skills/manage-evals/scripts/manage_evals.py compare \
    "swebench/litellm_proxy-claude-sonnet-4-5-20250929/23775164157/" \
    --auto-baseline \
    --post-comment --pr 2334 --repo OpenHands/software-agent-sdk

Available Data Per Run

Each run stores files at https://results.eval.all-hands.dev/{run_path}/:

Dashboard

The eval monitor dashboard provides a visual view of runs:

https://openhands-eval-monitor.vercel.app/?run={benchmark}/{model_slug}/{run_id}/

Interpreting Results

• Success rate = resolved / min(eval_limit, total_instances)
• A 50-instance sample has natural variance of ±2-4 resolved instances between runs
• Focus on instance-level changes (gained/lost) to understand regressions vs. noise
• If the same set of instances is resolved, the difference is likely noise

Additional Resources

Reference Files

• references/eval-infrastructure.md — Detailed documentation on the evaluation infrastructure, GCS paths, metadata format, and workflow triggers

Scripts

• scripts/manage_evals.py — Standalone comparison script with auto-baseline detection and GitHub comment posting