Nasde Benchmark Runner

NASDE Benchmark Runner

Run coding agent benchmarks with nasde and verify results. The two-stage pipeline: Harbor runs agents in Docker containers (functional test → reward 0/1), then an LLM-as-a-Judge scores architecture quality across multiple dimensions.

Authentication setup

Before running any benchmark, set up authentication tokens for the agents you plan to run. Both OS and auth method matter — pick the right command per row.

Step 1 — Ask the user which auth they prefer

Always ask the user before running, never assume. Two questions:

1. Which agents will you run? (Claude / Codex / Gemini, any combination)
2. For each agent, OAuth (subscription) or API key (per-token billing)? Default recommendation: OAuth where available — no per-token cost, no env vars to manage.

Then detect their OS and pick the matching script row from the table below. On Windows, also ask whether they're in PowerShell or WSL (cmd.exe is not directly supported — see "Windows: cmd.exe" below).

Where the auth scripts live

The OAuth scripts ship inside this skill. After nasde install-skills they are at:

• User scope (default): ~/.claude/skills/nasde-benchmark-runner/scripts/ (macOS/Linux/WSL) or %USERPROFILE%\.claude\skills\nasde-benchmark-runner\scripts\ (Windows PowerShell)
• Project scope: <project>/.claude/skills/nasde-benchmark-runner/scripts/ (if installed with nasde install-skills --scope project)
• Editable nasde checkout (devs only): <repo>/scripts/ — same files, mirrored from the skill bundle

Below, <SKILL_SCRIPTS> is shorthand for whichever absolute path applies. Resolve it once, then substitute it in every command. Verify the path with ls <SKILL_SCRIPTS> before telling the user to source anything — if the directory is missing, they need to run nasde install-skills first.

Step 2 — Run the right script per agent × OS

Priority order: Claude → Codex → Gemini. Claude is required even for non-Claude variants when [evaluation] backend = "claude" (default), because the assessment evaluator spawns claude CLI as a subprocess.

Claude Code

Prerequisite for OAuth: claude CLI installed and claude ran once to log in.

The script exports CLAUDE_CODE_OAUTH_TOKEN. This is required for both Claude variant runs AND assessment evaluation (when [evaluation] backend = "claude" — the default).

Codex

The OAuth scripts only validate ~/.codex/auth.json (or %USERPROFILE%\.codex\auth.json) — nasde then opts Harbor into uploading it into the sandbox (it sets CODEX_FORCE_AUTH_JSON=true when no API key is present but the file exists; harbor 0.13's Codex agent otherwise defaults to OPENAI_API_KEY and would write an empty key). API key always takes priority over OAuth when both are present.

Gemini CLI

The OAuth scripts export GEMINI_OAUTH_CREDS (the raw JSON) — ConfigurableGemini reads that env var and injects credentials into the sandbox. API key always takes priority over OAuth.

Combined setup for cross-agent runs

Resolve <SKILL_SCRIPTS> first, then run all three.

macOS / Linux / Windows WSL:

SKILL_SCRIPTS=~/.claude/skills/nasde-benchmark-runner/scripts   # adjust if --scope project
source $SKILL_SCRIPTS/export_oauth_token.sh         # Claude (subscription)
source $SKILL_SCRIPTS/export_codex_oauth_token.sh   # Codex (subscription) — or: export CODEX_API_KEY=...
source $SKILL_SCRIPTS/export_gemini_oauth_token.sh  # Gemini (Google account) — or: export GEMINI_API_KEY=...

Windows PowerShell:

$SkillScripts = "$env:USERPROFILE\.claude\skills\nasde-benchmark-runner\scripts"
. "$SkillScripts\export_oauth_token.ps1"
. "$SkillScripts\export_codex_oauth_token.ps1"
. "$SkillScripts\export_gemini_oauth_token.ps1"

Windows: cmd.exe

cmd.exe is not supported directly — .ps1 requires PowerShell, .sh requires bash. Two workarounds:

1. Open PowerShell (powershell.exe) and dot-source the .ps1 script. This is the simplest path on a vanilla Windows install.
2. Use WSL (wsl -d Ubuntu) and source the .sh script. This is the recommended path if you also want Docker Desktop with the WSL2 backend, which is the most common dev setup.

If a user is in cmd.exe, point them to one of these two — don't try to extract the token manually.

Important: chain auth into the run command

Exported env vars (CLAUDE_CODE_OAUTH_TOKEN, etc.) do not persist across separate shell invocations — every command in an automated runner (and every separate terminal call) is a fresh shell, so a lone source ... in one step is gone by the next. When running non-interactively, chain the source script(s) and the nasde call in a single command with &&:

# bash/zsh — both tokens chained into one invocation
source $SKILL_SCRIPTS/export_codex_oauth_token.sh && source $SKILL_SCRIPTS/export_oauth_token.sh && \
  nasde run --variant codex-vanilla --tasks my-task -C path/to/benchmark

Chain exactly the scripts the run needs: the agent's token (claude/codex/gemini) and the evaluator's token — the evaluator uses [evaluation] backend from nasde.toml, which defaults to claude, so a codex/gemini run with the default evaluator still needs export_oauth_token.sh too. In an interactive terminal session a one-time source in the same session is fine; the chaining rule matters for scripts, CI, and tool-driven runs.

Running benchmarks

All commands assume -C points to the benchmark project directory.

Basic run (all tasks, default variant)

nasde run -C path/to/benchmark

Assessment evaluation runs by default. This is the standard workflow.

Specific variant and tasks

# Single task, specific variant
nasde run --variant guided --tasks my-task -C path/to/benchmark

# Multiple tasks
nasde run --variant baseline --tasks task-a,task-b -C path/to/benchmark

With Opik tracing

nasde run --variant baseline --tasks my-task -C path/to/benchmark --with-opik

After this completes, ALWAYS verify Opik results (see Opik verification below).

Harbor only (skip assessment)

nasde run --variant baseline -C path/to/benchmark --without-eval

Parallel runs (multiple variants)

Do not use --all-variants when you want parallelism. --all-variants runs variants sequentially in a single process (one variant after another). To run two or more variants in parallel, launch separate nasde run processes with & and wait — each job directory gets a unique random suffix, so concurrent runs are collision-safe:

nasde run --variant vanilla --tasks my-task -C path/to/benchmark &
nasde run --variant guided --tasks my-task -C path/to/benchmark &
wait

Use --all-variants only when you want one variant after another (e.g. to limit total resource use, or when running Claude variants where parallel runs risk Docker OOM — see warning below).

For deterministic job names, use --job-suffix:

nasde run --variant vanilla --job-suffix run1 -C path/to/benchmark

Running Codex variants

Codex variants use AGENTS.md (instead of CLAUDE.md) and require either codex login (ChatGPT subscription) or CODEX_API_KEY/OPENAI_API_KEY (API billing).

CRITICAL: Codex model must be set explicitly. The nasde.toml default model (e.g. claude-sonnet-4-6) is designed for Claude and will be passed to Codex if not overridden. Codex CLI will silently accept invalid model names but produce garbage results (0% pass rate). Always set the model via --model flag or in variant.toml.

# Option A: ChatGPT subscription (no env vars needed after codex login)
nasde run --variant codex-vanilla --model gpt-5.3-codex -C path/to/benchmark

# Option B: API key
export $(grep CODEX_API_KEY .env)
nasde run --variant codex-vanilla --model gpt-5.3-codex -C path/to/benchmark

Codex models (recommended first, as of 2026-03):

• gpt-5.4 — flagship frontier model, best overall for professional work
• gpt-5.4-mini — fast, efficient mini model for responsive coding and subagents
• gpt-5.3-codex — industry-leading coding model for complex software engineering
• gpt-5.3-codex-spark — near-instant real-time coding iteration (ChatGPT Pro only)

Older alternatives: gpt-5.2-codex, gpt-5.1-codex, gpt-5-codex, gpt-5-codex-mini

Codex supports any model compatible with the Responses API. Chat Completions API support is deprecated.

Setting model in variant.toml (preferred over --model flag):

agent = "codex"
model = "gpt-5.4"

This avoids accidentally inheriting the Claude model from nasde.toml.

Running Gemini CLI variants

Gemini variants use GEMINI.md (instead of CLAUDE.md) and require either gemini login (Google account) or GEMINI_API_KEY/GOOGLE_API_KEY (API billing).

CRITICAL: Gemini model must use google/ prefix. Harbor requires model names in provider/model_name format. Always set via --model flag or in variant.toml.

# Option A: Google account OAuth (no env vars needed after gemini login)
nasde run --variant gemini-vanilla --model google/gemini-3-flash-preview -C path/to/benchmark

# Option B: API key
export GEMINI_API_KEY=your-key
nasde run --variant gemini-vanilla --model google/gemini-3-flash-preview -C path/to/benchmark

Gemini models (recommended first, as of 2026-03):

• google/gemini-3.1-pro-preview — advanced thinking model, deep reasoning
• google/gemini-3-flash-preview — best quality/speed ratio, daily coding
• google/gemini-3.1-flash-lite-preview — fastest, simple tasks

Setting model in variant.toml (preferred over --model flag):

agent = "gemini"
model = "google/gemini-3-flash-preview"

Cross-agent comparison (Claude vs Codex vs Gemini)

Set up all auth tokens first (see Authentication setup above), then run:

source scripts/export_oauth_token.sh && export $(grep CODEX_API_KEY .env) && source scripts/export_gemini_oauth_token.sh

# Run Claude, Codex, and Gemini variants — non-Claude MUST have --model override
nasde run --variant claude-vanilla -C path/to/benchmark --with-opik &
nasde run --variant codex-vanilla --model gpt-5.3-codex -C path/to/benchmark --with-opik &
nasde run --variant gemini-vanilla --model google/gemini-3-flash-preview -C path/to/benchmark --with-opik &
wait

WARNING: Running Claude variants in parallel can cause OOM (exit code 137) in Docker. Claude Code containers are memory-heavy (~2-4 GB each). Run Claude variants sequentially, or increase Docker Desktop memory to 16+ GB.

Custom model and timeout

nasde run --variant baseline --model claude-opus-4-7 --timeout 1200 -C path/to/benchmark

Timeout priority: --timeout flag overrides everything. Without it, Harbor uses task.toml [agent] timeout_sec per task. Timeouts are per-task only — there is no project-wide default in nasde.toml.

Re-evaluate existing results

nasde eval path/to/benchmark/jobs/2026-03-16__14-05-58 --with-opik -C path/to/benchmark

Token cost heuristic

Claude Code variants: When using CLAUDE_CODE_OAUTH_TOKEN (Claude subscription — no per-token cost):

• Run freely: total estimated time under 30 minutes (sum of tasks × variants)
• Ask first: over 30 minutes, OR when using ANTHROPIC_API_KEY (API billing)

Codex variants:

• ChatGPT OAuth (codex login): uses subscription credits, no per-token cost. Same heuristic as Claude subscription — run freely under 30 minutes.
• API key (CODEX_API_KEY): billed per-token. Always ask before running. Codex uses significantly more input tokens than Claude Code (~1M vs ~250K per task).

Gemini CLI variants:

• Google account OAuth (gemini login): free tier via Gemini Code Assist license (1M token context). Run freely.
• API key (GEMINI_API_KEY): billed per-token through Google AI Studio. Ask before running.
• Vertex AI (GOOGLE_API_KEY): billed through Google Cloud. Ask before running.

Task estimated times are derived from task.toml [agent] timeout_sec (timeout_sec / 60 is a rough upper bound; agents typically finish faster). When --tasks filters are used, count only selected tasks.

Viewing results

After a run, results are in jobs/<timestamp>/:

# Job summary
cat jobs/<timestamp>/result.json | python3 -m json.tool

# Per-trial results
cat jobs/<timestamp>/<trial-id>/result.json | python3 -m json.tool

# Verifier output (what test.sh printed)
cat jobs/<timestamp>/<trial-id>/verifier/test-stdout.txt

# Assessment scores
cat jobs/<timestamp>/<trial-id>/assessment_eval.json | python3 -m json.tool

Using Harbor CLI for viewing

nasde harbor view path/to/benchmark/jobs/<timestamp>

Comparing models — quality vs cost / tokens (PRIMARY method)

When the user asks "which model/agent is best?" or "which is most efficient?", the answer is a two-axis scatter (quality vs cost, quality vs tokens), not a single ranked number. Show the data honestly and let the reader judge — the convention here follows charts like Artificial Analysis's intelligence-vs-tokens plots: raw points with full names, a shaded "most attractive" region (high quality, low cost/tokens), and no verdict painted on individual points. Two reasons this is the primary method:

1. Quality and cost are two axes, and you keep both. Collapsing them into one number throws away information. Pareto dominance is the concept you reason with (point A is dominated if some B is no-worse on both axes and strictly better on one), but it is not a tag stamped on the chart — at small n a hard "dominated / never pick" label on a point with no variance over-claims. State dominance in prose when it is clear from the data; let the chart stay raw.
2. Position is invariant to where you put the score zero. That is exactly why nasde deliberately does not compute a scalar "token efficiency" or "cost efficiency" (score / denominator). See "Why no scalar efficiency" below.

The panels — one shared cost panel + one token panel per provider

• Quality × cost ($) — price-dependent, but one panel for all providers: USD is a common unit, so cross-provider comparison on cost is fair. Uses cost_usd (from pricing.toml).
• Quality × tokens — price-independent, but one panel per provider. Token counts are in each model's native tokenizer — Anthropic, OpenAI, and Google count tokens differently (verified: the same task is ~5.0M prompt tokens for Claude vs ~3.3M for a GPT model), so a "tokens" axis that mixes providers compares different units. The generator therefore splits the token view by provider (2 providers → 3 panels, 3 → 4). Compare token counts only within a provider (e.g. sonnet vs opus, or vanilla vs +skill on the same provider).

When the cost panel and the per-provider token panels tell the same story, the conclusion is stronger. A model can look token-light yet cost more (high per-token price) — that disagreement is itself a finding.

Future option (not implemented): re-tokenize every model's output with a single tokenizer (the Artificial Analysis approach — they re-encode all output with OpenAI's o200k_base) to get one cross-provider token axis. Our cost axis is already a fair common unit, so this is a nice-to-have, not a blocker; it would need the raw output text (the trajectory only stores counts) plus a tokenizer dependency.

Hard scoping rules — do NOT violate

Compare points only within:

• ONE task. Never aggregate across tasks of different difficulty into a single number. This is the paired-difference principle: report per-task deltas against a shared baseline, never a cross-task mean. An easy task and a hard task averaged together is a meaningless number.
• The same dimensions_fingerprint. A changed rubric (added/removed dimension, changed max_score, changed description) is a different benchmark and its scores are not comparable.
• The same reasoning_effort. Effort is part of the comparison axis. The toolkit groups economics by (agent_name, model_name, reasoning_effort). A trial's artifacts carry a reasoning_effort stamp (a string; empty "" means "not overridden — the Harbor family default"). gpt-5.3-codex at high and at low are two different points, not one.

If the points you are about to plot span more than one task, fingerprint, or effort, split them into separate charts — one Pareto chart per (task, fingerprint, effort) cell.

Sample size — n=1 is a signal, not a conclusion

• n=1 (a single trial) has no variance — it is a preliminary signal, never a conclusion. Label it as such.
• n≥2 gives the inter-trial std that the toolkit now reports (per model group). You need n≥2 before claiming a model "beats" another; a 0.02 score gap with no std is noise.

Source of truth

The raw numbers come straight from nasde results-export (see below). For each trial:

• score — normalized_score_mean in assessment_summary.json (or normalized_score / score).
• token_usage.output_tokens / token_usage.total_tokens — in metrics.json (and mirrored on the summary). Output tokens is the default token axis; total is available via --token-axis total.
• cost_usd — in metrics.json. null for an unpriced/legacy-trajectory model — such a point is dropped from the cost panel but still appears on the token panel.
• reasoning_effort, model_name, agent_name, task_name — stamped on the artifacts; the first three define the group, task_name enforces the one-task scope.

Export first, then compare — the export step is required, not optional: it is the single place that computes per-trial cost and token economics (from agent/trajectory.json + the price catalog; ADR-011) and flattens the nested jobs/<job>/<trial>/ tree into one dir per trial. A raw jobs/ dir has no metrics.json and is two levels deep, so the generator reads only exported dirs (or explicit --points) — never raw jobs/.

nasde results-export path/to/benchmark/jobs/<timestamp> --to /tmp/myexport -C path/to/benchmark

Generating the chart

The skill ships a reference generator at <SKILL_SCRIPTS>/pareto.py (matplotlib + stdlib only — install matplotlib into whatever Python you run it with, e.g. uv run --with matplotlib python <SKILL_SCRIPTS>/pareto.py ...). It reads an export dir directly, or accepts explicit data points, and draws the panels in the raw-points style (a green shaded region marks the most attractive corner — high quality, low cost/tokens — no front line, no dominated tags). Visual encoding for the skill×model matrix: color = provider, marker shape = skill (circle = vanilla, a distinct shape per skill — assigned stably, so the same skill keeps its shape across providers and every panel), and a thin line links the variants of one model, so the quality/cost shift from adding a skill to a given model is visible at a glance. The model name is labelled once per model — at its lowest point (the variant with the lowest score) — so a model with several variants is not labelled repeatedly; the connecting line and marker shape identify the other variants. A single shared encoding legend sits to the right of all panels (provider colors + a shape-per-skill list with the real skill names). The shape palette holds ~10 skills; beyond that shapes repeat and the generator prints a warning (read the labels/legend to disambiguate) — it never collides shapes silently.

# From an export dir (one subdir per trial). --task scopes a multi-task export to one task.
# Title MUST state the scope (task, fingerprint, effort).
uv run --with matplotlib python <SKILL_SCRIPTS>/pareto.py \
  --export-dir /path/to/nasde-results \
  --task ddd-weather-discount \
  --title "weather-discount — fp=abc123def456 — effort=default" \
  --out /tmp/quality_chart.png

# Or explicit points: name,effort,score,cost_usd,output_tokens (cost may be empty for unpriced).
uv run --with matplotlib python <SKILL_SCRIPTS>/pareto.py \
  --point "claude-opus-4-8,,0.92,26.30,69055" \
  --point "claude-sonnet-4-6,,0.80,8.55,33430" \
  --out /tmp/quality_chart.png

--token-axis {output,total} picks the token panel's x-axis (default output, log scale). Trials are grouped by (agent_name, model_name, reasoning_effort) — the same key the toolkit uses for run-summary economics — so two variants of the same model (e.g. claude-vanilla vs claude-ntcoding-tactical-ddd, both on claude-sonnet-4-6) stay separate points, not one averaged blob. Each group's per-axis std is printed to stdout (n≥2), with an [n=1 preliminary signal] flag otherwise. (With --point you have no separate agent field, so the point name doubles as the variant.)

Chart rule — full model version strings, always

• Never use bare family names ("sonnet", "opus", "gpt") in labels or legends. Always the full version string: claude-sonnet-4-6, claude-opus-4-8, gpt-5.3-codex, google/gemini-3-flash-preview.
• The reasoning effort must be visible on the chart/legend too (the generator prints effort=<value> per point; default when the stamp is empty).

Why no scalar "efficiency"

It is tempting to collapse the picture into one number — efficiency = score / cost or score / tokens — and rank by it. nasde deliberately does not, for two reasons:

1. It is lossy. It crushes a 2D trade-off (quality vs cost) into one scalar, hiding which axis a model wins on. A buyer choosing between "cheap and decent" and "expensive and excellent" needs both numbers, not their ratio.
2. It has an arbitrary zero. score is a normalized rubric score whose zero is "empty rubric" — an unreachable, arbitrary reference point. Because the ratio divides by a denominator from that arbitrary zero, the ranking is not invariant to a baseline shift: the same trials can produce a different "winner" depending only on where you place the score zero. A Pareto front does not move when you shift the score axis, so it is the honest comparison. (This is a locked design decision — do not reintroduce a scalar efficiency.)

Optional secondary helper — baseline-relative Δscore

For a single task, it is fine to report each variant's Δscore against the vanilla baseline on that same task (score_variant − score_vanilla). That is a paired difference against a shared reference and is legitimate. What is not legitimate is a scalar score/$ measured from zero — that is the arbitrary-zero ratio above. Keep Δscore as a supporting view; the Pareto front stays primary.

Opik verification

After every run with --with-opik, verify results via REST API. This is mandatory — Opik has known issues with long-running trials where data may not arrive completely.

Verification script

Use Python urllib.request (never curl — it drops the Comet-Workspace header):

python3 -c "
import urllib.request, json

req = urllib.request.Request(
    'https://www.comet.com/opik/api/v1/private/traces?project_name=<PROJECT_NAME>&limit=1',
    headers={
        'authorization': '<OPIK_API_KEY>',
        'Comet-Workspace': '<OPIK_WORKSPACE>',
    },
)
resp = json.loads(urllib.request.urlopen(req).read())
trace = resp['content'][0]
print(f'Trace: {trace[\"name\"]}')
print(f'ID:    {trace[\"id\"]}')
print()
for s in sorted(trace.get('feedback_scores', []), key=lambda x: x['name']):
    print(f'  {s[\"name\"]}: {s[\"value\"]}')
"

Credentials are in the benchmark's .env file (or parent directory).

What to check

For each trace, verify:

1. Feedback scores present — should include:
   • reward (from Harbor verifier: 0.0 or 1.0)
   • duration_sec (trial execution time)
   • arch_<dimension> for each assessment dimension (normalized 0.0-1.0)
   • arch_total (overall normalized score)
2. Trace name format: <agent-name>/<trial-name> (e.g. baseline/ddd-threshold-discount__4tTaKwg)
3. Tags: should include harbor and the variant name

Finding trace IDs

The trace ID is printed during the run:

OPIK: Started logging traces to ... ?trace_id=<TRACE_ID>&...

Or search by project:

import opik
client = opik.Opik()
traces = client.search_traces(
    project_name='<benchmark-name>',
    filter_string='name = "<agent-name>/<trial-name>"',
    max_results=1,
    wait_for_at_least=1,
    wait_for_timeout=10,
)

Troubleshooting

"No module named 'evals'" or import errors

Harbor resolves import_path from harbor_config.json via importlib. If using the built-in agent (nasde_toolkit.agents.configurable_claude:ConfigurableClaude), this works when nasde-toolkit is installed. For custom agents, ensure the module is on sys.path.

Docker build fails

Test the Dockerfile independently:

docker build -t test-env -f tasks/<task>/environment/Dockerfile .
docker run --rm -it test-env bash

Assessment eval fails with "No artifacts/workspace/"

Harbor didn't copy artifacts. Check:

• The artifacts config in the merged Harbor config (source path must match container layout)
• The trial log: jobs/<ts>/<trial>/trial.log

Opik scores missing after --with-opik

1. Check trace exists: use the REST API verification script above
2. If trace exists but no arch_* scores: assessment eval didn't run or failed. Check the CLI output for errors.
3. If no trace at all: Opik tracking wasn't enabled. Verify .env has OPIK_API_KEY and OPIK_WORKSPACE.

Codex trial fails immediately (reward 0, 0/100)

Check the agent log for errors:

head -20 jobs/<ts>/<trial>/agent/codex.txt

Common causes:

• Incorrect API key provided: '' — no auth configured. Either run codex login (ChatGPT subscription) or set CODEX_API_KEY: export $(grep CODEX_API_KEY .env)
• model 'X' does not exist — wrong model name. Use gpt-5.4, gpt-5.4-mini, gpt-5.3-codex, or gpt-5.3-codex-spark
• 0% pass rate, low scores, but trials completed — likely inherited Claude model name (e.g. claude-sonnet-4-6) instead of OpenAI model. Check config.json in the job dir for model_name. Fix by adding model = "gpt-5.3-codex" to variant.toml or using --model gpt-5.3-codex
• Tool 'web_search_preview' is not supported — model doesn't support Codex tools
• Model metadata for 'X' not found — warning only, usually followed by the real error

Gemini trial fails immediately (reward 0, 0/100)

Check the agent log for errors:

head -20 jobs/<ts>/<trial>/agent/gemini-cli.txt

Common causes:

• GEMINI_API_KEY is not set — no auth configured. Either run gemini login or set GEMINI_API_KEY
• Model name must be in the format provider/model_name — model must include google/ prefix (e.g. google/gemini-3-flash-preview)
• Node.js errors — Gemini CLI requires Node.js 22+. Check the Docker image includes nvm setup
• DNS resolution failures — cloud sandboxes may not resolve generativelanguage.googleapis.com. ConfigurableGemini auto-fixes this, but custom configs may not

Trial reward is 0 but code looks correct

Read the verifier output:

cat jobs/<ts>/<trial>/verifier/test-stdout.txt

This shows exactly which step in test.sh failed.