Genbench

genbench

One command, from the repo root (ANTHROPIC_API_KEY in the environment):

pnpm build                                      # genbench reads the built @vendoai/* dists
pnpm genbench run --prompt <case-id>            # one case, all contenders, opens preview.html
pnpm genbench run                               # every case in one world
pnpm genbench run --models opus,sonnet,haiku    # expand the harness x model matrix
pnpm genbench run --world maple                 # choose world (default: maple)
pnpm genbench run --world all                   # every world into ONE run folder

The corpus is 14 worlds, 196 cases — 15 per world, except buildlog and fieldops at 10 and logistics, observability, product-analytics and trades-accounting at 14. Case ids in maple: spend-overview, spend-chart, pending-transfers, account-balances, no-pending-transfers, transfer-receipt, cancel-both-pending, transfer-activity-feed, room-for-dana, rent-check, bills-calendar, money-dashboard, dining-budget-cap, category-groups, savings-goals.

• Contenders: vendo (real pipeline, this working tree) · diy (one raw streamText call) · claude-code (stock Agent SDK in a scratch dir). Byte-identical world info per contender: ONE serializer, worldBlock in genbench/src/vendo.ts, enforced by the fairness test in genbench/tests/diy.test.ts against what the vendo driver really receives.
• Worlds: genbench/worlds/<name>/{world.json, cases.json} plus an optional font.woff2 the harness injects into every contender's page. world.json = tools + canned data + theme + style rubric; cases.json = prompt + pass lines. Conventions: money in cents; a tool with data is a read, takes-only is a write.
• Output: genbench/runs/<run>/<contender>/<case>/{artifact.tsx (vendo only), page.html, screenshot.png, result.json} + summary.json (the run's ONE aggregate, per column) + preview.html (live embedded screens, world-data panel, live tool-call feed). runs/ is gitignored. <contender> is the column slug <harness>-<model> — vendo-sonnet, diy-opus, claude-code-haiku. Under --world all the case folder is <world>/<case>, because two worlds ship the same case id.
• Floor checks are deterministic (delivered / renders / valid / honestData / wiredActions via click-probe). A check with nothing in front of it is VACUOUS and one whose grader was unreachable is DEGRADED; neither scores, in either direction. The pass lines are the rubric a pinned judge (versioned rubric contract) grades on every run — any edit to its prompt bumps rubricVersion and resets comparability.
• Exit code: any floor failure exits 1; a judge outage or a failed rubric line does not. The last stdout line says which — floor failures: 2 (exit 1). Through pnpm that is followed by pnpm's own ELIFECYCLE line, which is not a second failure.
• Judge spend is reported separately — judged.cost in result.json and one line under the preview's run header. It is NEVER folded into a contender's cost, which is only what that contender spent building its screen.
• Budgets are per contender: five minutes for vendo and diy, twelve for claude-code, which runs its own ten-minute wall clock inside the driver.
• Rough cost: one case ≈ 1-4 min ≈ $0.30-$0.50 + judge; one world is 10-15x that and --world all is 196x; --models multiplies by the model count. Prices in src/meter.ts are as of 2026-08-08 (Sonnet 5 is on intro pricing through 2026-08-31) — token counts are the durable number, dollars are not.
• --prompt runs open preview.html on macOS; full runs just print the path. CI or GENBENCH_NO_OPEN=1 suppresses the window.
• Gotchas: test genbench with pnpm --filter @vendoai/genbench test (its vitest.config.ts caps the pool at 1-2 workers) — never the full repo suite, and not a bare npx vitest, which resolves a different vitest that rejects this repo's worker flags. The two money-spending tests (judge smoke, claude-code driver) need GENBENCH_LIVE=1 and ANTHROPIC_API_KEY; both stay skipped otherwise. --lane build is deferred and says so.