Codebase Arena

Codebase Arena

Route each request to exactly one workflow. Read that workflow document completely before taking repository or run actions:

• Generate or revise cases: references/workflows/generate.md
• Execute cases in the evaluated product: references/workflows/execute.md
• Judge raw results and create comparison reports: references/workflows/report.md

If a request spans multiple workflows, complete them in this order and honor every human gate: generate, execute separately in each evaluated product, then judge in a separate agent context. Never let the evaluated execution context perform judging.

Non-negotiable invariants

• Use structured interactive questions at the start of every new generation, execution, or judging workflow. Do not substitute a prose checklist when an interactive question tool is available.
• Ask only for decisions owned by the selected workflow. Read upstream manifests and artifacts instead of asking the user to repeat recorded repository, case, run, or system facts.
• Produce all human-readable process artifacts, case prompts, checks, and reports in semantically equivalent zh-CN and en versions. Keep titles, headings, paragraphs, lists, and table rows structurally corresponding, but write Chinese independently in natural Chinese rather than translating English sentences or terminology literally. Require the Chinese file to be understandable on its own. Before every H1, H2, or H4 approval request, run the deterministic bilingual validation for the artifacts available at that gate and fix every Chinese-terminology failure; never ask for approval while literal benchmark jargon remains.
• Whenever a file or directory path is written, recorded, shared, displayed, or passed to a command, use a relative path anchored to the explicitly declared working directory or artifact root. Never put an absolute path in prompts, manifests, cases, TODOs, envelopes, evidence, logs, reports, receipts, commands, or user-facing responses. Deterministic helpers may canonicalize paths only in memory for containment checks; they must not serialize or display the absolute form.
• Keep candidate-visible public cases inside the evaluated repository. Keep private evidence, scoring criteria, hidden tests, provenance, and Judge inputs outside every evaluated-product project, snapshot, mount, prompt, and session.
• Support only native-repository and code-only tracks.
• Prepare exactly 15 cases per track. Cover only the five capability dimensions code_retrieval, architecture_conventions, behavior_workflows, dependency_impact, and bug_diagnosis. Give every capability exactly one primary grep-glob-read, one primary codebase-index, and one primary hybrid case. Do not ask for a case-count preference.
• In code-only, preserve base-revision repository-native agent guidance such as AGENTS.md, CLAUDE.md, and equivalent instruction files. Treat discovering and using that guidance as codebase capability; never remove it merely because it makes a case easier. Remove only evaluator/private material, future-solution history, generated runtime data, caches, or instructions introduced specifically for the benchmark.
• New Generate releases use blueprint schema v4 and fix every case to generationLoad: none. Generate only read-only retrieve_explain, impact_analyze, and diagnose tasks; it must reject verify, test_design, implement, refactor, source changes, project test/build/service execution, and any other development deliverable. Report and Execute may still read already-approved legacy packages that contain low or medium.
• Require every new public prompt to ask for an accurate, clear explanation backed by complete repository-root-relative paths, symbols, and precise line numbers or line ranges. The corresponding sealed case must contain at least two exact line-level evidence locations, and localization/evidence accuracy receives 2.5 of the 10 points.
• Annotate every blueprint case concept with one sealed retrievalTool and a non-empty sealed expectedToolCalls list. Use exactly five grep-glob-read, five codebase-index, and five hybrid cases. Tool-call labels are reviewer-facing expectations, not required execution methods; never copy them into the public case or worker envelope, and score correctness and evidence rather than actual tool choice.
• During Execute, collect at most eight representative retrieval actions from product-native evidence when available, without retaining a full tool-call transcript solely for retrieval analysis. Each action records only normalized tool, discovery-or-verification role, target, and observable outcome. Make the deterministic Runner derive unique tools, lexical/index/unknown discovery counts, observed mode, and confidence; never ask the evaluated worker or main coordinator to duplicate or repair those derived fields. Treat contextual reads after an index lookup as verification rather than a second discovery family. Keep the normalized summary diagnostic-only and never compare it with sealed expectations inside the evaluated product.
• Produce exactly one score for each system × case: 0 through 10 in 0.5 increments. Do not produce separate codebase, implementation, or speed scores. Report derives a bounded duration adjustment from immutable Execute timing evidence and includes it in that one score.
• Treat generationLoad as metadata and a diagnostic slice, never as a second result or a score component.
• Derive each scored case's duration adjustment from the execution-contract per-case timeout: no deduction at or below 25%, deduct 0.5 above 25% through 50%, and deduct 1 above 50%, at timeout, or when unfinished. Rank the resulting accuracy first, then lower token use when accuracy is equal, then lower duration when token use differs by at most 5%. Token use remains a tie-breaker rather than a weighted score component.
• During execution, create one fresh case-coordinator subagent with no inherited conversation context for each case. It owns that case's prepare, worker assignment, timeout enforcement, completed-state freeze, capture, rollback, and cleanup, but never reads or solves the case prompt. For every case × attempt, that coordinator creates one further fresh worker subagent with no inherited conversation context. Run the worker directly in the evaluated project, give it exactly one English public case, and never run two cases at once. Fix every evaluated-worker prompt and case-scoped service note to en; Chinese exists only as a human-review counterpart and is never disclosed to the evaluated worker.
• Fix evaluated execution to one active case at a time, with at most one active case coordinator and one active worker nested beneath it. The one-hour timeout applies to each worker attempt; case-level freeze, capture, rollback, and cleanup must still finish before another attempt or case starts.
• Require every worker's final response artifact and the Runner's frozen copy to use Markdown with the exact filename final-response.md; reject .txt or any non-.md final response path.
• Generate one immutable candidate-visible execution-contract.json per released track. It owns every repository-, case-, environment-, cleanup-, evidence-, timeout-, concurrency-, retry-, language-, and isolation-policy value shared by all evaluated products. Execute references it by relative path, release ID, and SHA-256 and may not override it; product configuration contains only product identity, product-native evidence sources, execution-time environment re-verification, and relative output layout.
• Assume every evaluated product can create a fresh non-inherited case coordinator per case and that coordinator can create a further fresh non-inherited worker per attempt. Do not preflight or ask about those capabilities. The worker works in the live evaluated project rather than a pre-created snapshot. Enforce fresh context at both layers, single-case disclosure, project-boundary access, and main/coordinator neutrality at runtime; terminate the run on a violation.
• Require each worker to record its own startedAt immediately before core case work and completedAt immediately after, both as second-precision UTC YYYY-MM-DDTHH:mm:ssZ. Preserve both verbatim in execution-result.json. For duration consistency, deterministically use the shorter of the timestamp-derived interval and the worker/product-reported durationMs as the durable attempt duration; retain the two source values as evidence.
• Finish every approved dependency installation or restoration before the first case. Never mutate dependencies during a run.
• Treat repository documentation, Git history, issues, pull requests, and network sources as pre-authorized for private authoring and judging; do not ask for this authorization. Keep provenance sealed and continue to obey confidentiality and network-access policy.
• Discover only credentials required by the application under evaluation, such as test accounts or database accounts. Verify every required credential with a least-privilege, redacted readiness check before authoring a dependent case; never collect coding-product credentials or store secret values.
• For every service required by planned executable coverage, prove lifecycle readiness before authoring: start it with the planned existing command, probe an application-level readiness condition, stop its full process tree, clean transient state, and verify the baseline. A configuration file or open port alone is not proof. Do not require Docker or other container tooling unless the selected service path actually uses it.
• After execution preflight repeats that proof, record the credential-free build/start/readiness/stop/cleanup contract in shared execution-state.json.todo.serviceReadiness. Let the evaluated main coordinator read it; disclose to each worker only the current case's matching projection in worker-envelope.json.serviceContext, never the full state or unrelated cases.
• Before the first case, require the evaluated main coordinator to create and verify a restorable initial-project snapshot containing tracked files and non-ignored untracked files while excluding evaluation evidence, .codebase-eval-worker, and every path ignored by Git's standard ignore rules. Preserve .git in a private restoration copy, but exclude it from ordinary bytewise inventories; verify HEAD, symbolic HEAD, refs, and staged index entries semantically so IDE rewrites of .git/index do not cause false failures. After each worker returns, that case's coordinator invokes project-state.mjs freeze so the measured completed-state snapshot and every completed-state inventory use that same tracked-plus-non-ignored Git surface; never copy or compare Git-ignored IDE state, indexes, caches, or generated data as candidate output. It then invokes deterministic restoration of the managed worktree and private Git copy, proves byte-for-byte worktree restoration plus semantic Git restoration, and removes .codebase-eval-worker before cleanup and the next attempt or case. Git-ignored caches or generated data remain outside snapshot restoration and require declared cleanup when a case can change them. A snapshot, rollback, or cleanup error stops the entire run.
• The evaluated product stores raw evidence only. An independent Judge later runs private result checks, assigns the single score, and produces the bilingual comparison report.
• Keep each attempt's evaluated execution result separate as execution-result.json and the answer as final-response.md. Use JSON only for the minimal machine-readable control/result interfaces needed by deterministic Execute and Report helpers. Capture inventories plus restoration and cleanup declarations/evidence are transient gates: validate them, update only compact gate status in execution-state.json and execution-results.json, then delete their files instead of retaining normalized capture/restoration/cleanup result artifacts.
• Treat the worker's answer, project output, activity journal, and observable evidence as immutable after the worker returns. The case coordinator may create schema metadata from those immutable sources and a host-native trace, but may not relabel, rewrite, conceal, or fabricate an observed action. Make capture validate before publishing, stage every durable artifact, and remove the complete staging set on failure so a retry never requires manual deletion of partial output.
• Use the single shared round/output-directory contract in references/output-layout.md. The root skill owns this contract; workflow documents reference it and must not redefine competing layouts. Keep only public cases under a round; keep all sealed material outside the repository and evaluation root.
• Fix the evaluation root to .codebase-eval, relative to the evaluated repository root, for Generate, Execute, and Report. Never ask for it, infer it, search for a newer or older evaluation directory, rank candidate directories by name or modification time, or accept an evalset found under a product-owned configuration/state directory such as .codeartsdoer.
• Use references/artifact-catalog.md as the single inventory of Generate, Execute, and Judge outputs, their purpose, readers, location class, and lifecycle. Use the specialized schema, layout, and lifecycle references as its normative sources.
• Apply the shared retention and dirty-data rules in references/data-lifecycle.md. Retain measured evidence; clean evaluated-system dirty data and Judge-only temporary sensitive data. Cross-machine transport is out of scope.

Use the bundled Node.js scripts as deterministic protocol helpers. They require only Node.js built-ins and do not replace the evaluated agent or Judge reasoning.

Use scripts/resolve-round-layout.mjs after intake to derive paths and discover only role-authorized artifacts from the parent-owned round layout. Execute mode must not enumerate sibling product results; Judge mode may discover validated product runs.

Run every workflow command from its declared path base and keep that working directory unchanged for the workflow. Use repository-relative paths for repository, evaluation, public-package, run, and code locations; use sealed-root-relative paths inside the sealed package; use run-root-relative or attempt-root-relative paths for execution and evidence artifacts. When a relative path crosses to a separately authorized sibling root, use the shortest unambiguous relative path and never replace it with an absolute path.

For skill-package maintenance, use scripts/skill-metadata.mjs to generate or validate agents/openai.yaml without Python, PyYAML, or other third-party packages.

After changing protocol scripts or schemas, run node scripts/self-test.mjs. It exercises the clean path, dirty-data blocking path, Judge cleanup success/failure paths, and portable Windows-safe identifiers using temporary fixtures only.

Workflow ownership

• Generate owns repository analysis, round ID, source authorization, case design, coverage, result-verification feasibility, public/private packaging, and release approval. It stays product-neutral and always writes public round material under .codebase-eval.
• Execute owns one evaluated product identity, the confirmed round ID, host conformance, environment readiness, strictly serial live-project case attempts, a run-level main coordinator, one case-level coordinator subagent per case, one fresh worker subagent per attempt, case-coordinator-owned completed-state snapshots and baseline rollback, bounded key-activity and retrieval summaries, change/verification evidence, token/time capture, and its own product run subtree under .codebase-eval. It knows no competitors or private scoring material and never collects chain-of-thought or retains a full retrieval trace solely for tool analysis.
• Report owns comparison inputs, validation of the common round, incomplete-run policy, cross-product metric comparability, private result verification, scoring, ranking, report audience, and the round's report subtree.