Archestra Dev Bench Analysis

Archestra bench trajectory analysis

Map-reduce a finished archestra-bench run into a recommendations report, using Claude subagents for the judgment. The deterministic half (render + metrics + manifest) is done by the Rust archestra-bench prepare subcommand, so this skill reuses the analyzer's exact rendering, metrics, and ordering — it does not re-implement them. Mirrors ai-labs/analyzer (map = per-rollout triage; reduce = repo-grounded Tier-1/Tier-2 report); see ai-labs/analyzer/README.md.

The subagent fan-out runs through the native Workflow tool — two scripts under this skill's workflows/ directory drive the map and crawl phases, so the orchestrator never hand-batches Agent calls and the per-rollout triages never flow through its context. Calling those scripts here is your explicit opt-in to Workflow. The exact map/reduce prompt text lives in reference/prompts.md (this skill's directory) — bin/prepare.sh extracts the MAP block automatically; you still read the REDUCE sections verbatim in step 4. Do not paraphrase those prompts.

<SKILL_DIR> below is this skill's absolute directory (the one containing this file).

1. Prepare (deterministic: dir resolution + Rust prepare + arg shaping)

Run the helper (from anywhere — it derives the repo root). Pass a run dir, or omit it to pick the newest under ai-labs/experiments/:

<SKILL_DIR>/bin/prepare.sh "$ARGUMENTS"

It resolves the run dir to an absolute path, runs archestra-bench prepare (failures-first manifest; fail-fast — if it exits non-zero, surface the printed path:line error and stop), and writes under <RUN_DIR>/_prep_claude/: manifest.json, metrics.md, order.tsv (idx<TAB>id<TAB>outcome, manifest order), one pre-rendered triage prompt per rollout under prompts/<NN>.txt, and map-args.json (small, ready-to-pass args for the map workflow). It prints a KEY=value summary — capture RUN_DIR, TS, TRIAGE_DIR, MAP_ARGS, ANALYSES_DOC, REPORT_DOC — then the metrics block. State which RUN_DIR it chose.

2. Map — one triage workflow

Read the MAP_ARGS file (map-args.json) and pass its JSON as args to the Workflow tool with scriptPath: <SKILL_DIR>/workflows/map.mjs. The file is small by design — triageDir, promptsDir, and the rollouts array ({idx,id}, manifest order); the filled triage prompts live on disk under promptsDir, so the bulk never flows through your context. The workflow fans out one Sonnet triage agent per rollout (auto-batched at the concurrency cap — no manual 8-at-a-time loop); each reads its pre-rendered prompt (<promptsDir>/<NN>.txt) and the trajectory it points to, then Writes its judgment — a single JSON object (rubric grades, reward-hacking flag, observations) — to <TRIAGE_DIR>/<NN>.json (zero-padded idx). It returns {written, total}; if written < total, step 3's validator will name the missing indices. Sonnet is deliberate — triage is a cheap bounded read; reserve the stronger model for the reduce synthesis.

3. Validate + assemble (deterministic, in this loop)

Run the renderer — it validates every <TRIAGE_DIR>/<NN>.json against order.tsv, stamps rollout/outcome from order.tsv (never from model output), writes the rubrics JSONL (<RUN_DIR>/trajectory_rubrics_claude_<TS>.jsonl) and the analyses doc (truncation parity with the Rust analyzer included), and prints both paths as KEY=value:

node <SKILL_DIR>/bin/render-triage.mjs <RUN_DIR> <TS>

If it exits non-zero, it lists the invalid/missing indices (and any extra/stale files) on stderr: delete the extra files, re-run the map workflow with a rollouts subset containing only those indices — filter map-args.json with jq, e.g. jq '.rollouts |= map(select(.idx == 3 or .idx == 7))' <MAP_ARGS> — then re-run render-triage.mjs. Do this before the reduce step — a reduce failure must never discard the map work.

4. Reduce — repo-grounded report

Read the analyses doc (ANALYSES_DOC). Adopt the REDUCE system guidance from reference/prompts.md as your framing, then carry out the REDUCE task message (fill {ANALYSES_DOC_PATH} with ANALYSES_DOC, {RUN_DIR} absolute, {BACKEND_LOG_PATHS} with <RUN_DIR>/*.backend.log).

Ground every finding in file:line across platform/ (Tier 1) and ai-labs/ (Tier 2) by fanning the crawlers out through the crawl workflow: derive one issue/subsystem per cluster from the analyses doc, then call the Workflow tool with scriptPath: <SKILL_DIR>/workflows/crawl.mjs and args:

{
  "repoRoot": "<repo root absolute>",
  "crawlerSystem": "<the verbatim REDUCE crawler system prompt from reference/prompts.md>",
  "issues": [ { "label": "run_command-target", "prompt": "<one issue to investigate>" }, ... ]
}

It returns [{ label, evidence }] (.filter(Boolean) it — a hard-failing crawler yields a null) — the grounding you synthesize the report from. Before citing a surprising map claim, open the rollout's raw trajectory.md and confirm it.

Hard rule: the backend-log files (<RUN_DIR>/*.backend.log, e.g. basic.backend.log) are ~tens of MB. Never Read/cat them (yours or a subagent's) — only capped grep, e.g. grep -n -m 50 -F '<pattern>' <RUN_DIR>/basic.backend.log.

Write the report to REPORT_DOC (<RUN_DIR>/trajectory_analysis_claude_<TS>.md).

5. Report

Tell the user the output paths (analyses doc, rubrics JSONL, report) and a one-line headline (overall pass rate + the top Tier-1 finding).