]` 2. Use the Edit tool — `old_string` = **Current** text verbatim, `new_string` = **Proposed** text 3. If **Current** is `"none"` (new insertion): find the section header and insert the **Proposed** text after the last item in that block 4. Skip if **Current** text is not found verbatim → print `⚠ Skipped — current text not found` 5. Skip if **Proposed** text is already present → print `✓ Already applied — skipped` After processing all changes return **only** this compact JSON: `{"target":"","applied":N,"skipped":N}` ______________________________________________________________________ After all subagents complete, collect their JSON results and print the final summary: ``` ## Fix Apply — | Target | File | Applied | Skipped | |-------------|-------------------------------|---------|---------| | sw-engineer | .claude/agents/sw-engineer.md | 2 | 0 | → Run /calibrate to verify improvement. ``` Mark "Apply findings" completed. End your response with a `## Confidence` block per CLAUDE.md output standards. - **Timeout handling**: phase and pipeline budgets (see the constants block) prevent nested subagent hangs from cascading. Extension is granted once if the pipeline explains the delay in its output file — a second unexplained stall still triggers the cutoff. Timed-out pipelines appear with ⏱ prefix and `verdict:"timed_out"`; re-run individually with `/calibrate fast` after the session. - **Context safety**: each target runs in its own pipeline subagent — only a compact JSON (~200 bytes) returns to the main context. `all full ab` with 14 targets returns ~2.8KB total, well within limits. - **Scorer delegation**: Phase 3 delegates scoring to per-problem `general-purpose` subagents. Each scorer reads response files from disk, returns ~200 bytes. The pipeline subagent holds only compact JSONs regardless of N or A/B mode — no context budget concern. - **Nesting depth**: main → pipeline subagent → target/scorer agents (2 levels). Pipeline spawns both target agents (Phase 2) and scorer agents (Phase 3) at the same depth — no additional nesting. - `general-purpose` is a Claude Code built-in agent type (no `.claude/agents/general-purpose.md` file needed) — it provides a baseline Claude instance with access to all tools but no custom system prompt. - **Quasi-ground-truth limitation**: problems are generated by Claude — the same model family as the agents under test. A truly adversarial benchmark requires expert-authored problems. This benchmark reliably catches systematic blind spots and calibration drift even with this limitation. - **Calibration bias is the key signal**: positive bias (overconfident) → raise the agent's effective re-run threshold in MEMORY.md. Negative bias (underconfident) → confidence is conservative, no action needed. Near-zero → confidence is trustworthy. - **Do NOT use real project files**: benchmark only against synthetic inputs — no sensitive data and real files have no ground truth. - **Skill benchmarks** run the skill as a subagent against synthetic config or code; scored identically to agent benchmarks. - **Improvement loop**: systematic gaps → `` | consistent low recall → consider model tier upgrade (sonnet → opus) | large calibration bias → document adjusted threshold in MEMORY.md | re-calibrate after instruction changes to quantify improvement. - **Report always**: every invocation surfaces a report — benchmark runs print the new results table; bare `apply` (no `fast`/`full`) prints the saved report from the last run before applying, so the user always sees the basis for any changes before files are touched. - **`apply` semantics**: `fast apply` / `full apply` = run fresh benchmark then auto-apply the new proposals in one go. `apply` alone (no `fast`/`full`) = apply proposals from the most recent past run without re-running the benchmark. - **Stale proposals**: `apply` uses verbatim text matching (`old_string` = **Current** from proposal). If the agent file was edited between the benchmark run and `apply`, any change whose **Current** text no longer matches is skipped with a warning — no silent clobbering of intermediate edits. - **`routing` target vs `/audit` Check 12**: `/audit` Check 12 performs static analysis of description overlap (finds potential confusion zones); `/calibrate routing` tests behavioral impact — it generates real routing decisions and measures whether descriptions actually disambiguate. Run in sequence: `/audit` first (fast, structural), then `/calibrate routing` (behavioral, slower). They are complementary, not redundant. - **`routing` not in `all`**: routing tests orchestrator dispatch logic, not agent quality — excluded from batch calibration. Run `/calibrate routing` explicitly after any agent description change. - **Routing proposals**: the routing pipeline's Phase 5 writes description improvement proposals to `.claude/calibrate/runs//routing/report.md` — look in the Proposals section for targeted wording suggestions per confused pair. - Follow-up chains: - Recall < 0.70 or borderline → `/calibrate fast apply` → `/calibrate ` to verify improvement — stop and escalate to user if recall is still < 0.70 after this cycle (max 1 apply cycle per run) - Calibration bias > 0.15 → add adjusted threshold to MEMORY.md → note in next audit - Routing accuracy < 0.90 or hard accuracy < 0.80 → update descriptions for confused pairs → `/calibrate routing` to verify improvement - Recommended cadence: run before and after any significant agent instruction change; run `/calibrate routing` after any agent description change - **Internal Quality Loop suppressed during benchmarking**: the Phase 2 prompt explicitly tells target agents not to self-review before answering. This ensures calibration measures raw instruction quality — not the `(agent + loop)` composite. If the loop were enabled, it would inflate both recall and confidence by an unknown ratio, masking real instruction gaps and making it impossible to attribute improvement to instruction changes vs. the loop self-correcting at inference time. - **Skill-creator complement**: `/calibrate` benchmarks agents and skills via synthetic ground-truth problems; the official `skill-creator` from the anthropics/skills repository handles skill-level eval — trigger accuracy, A/B description testing, and description optimization. The two are complementary: run `/calibrate` for quality and recall, `skill-creator` for trigger reliability. - **A/B mode rationale**: every specialized agent adds system-prompt tokens — if a `general-purpose` subagent matches its recall and F1, the specialization adds no value. `ab` mode quantifies this gap per-target so you can decide whether to keep, retrain, or retire an agent. `significant` (Δ>0.10) confirms the agent's domain depth earns its cost; `marginal` (0.05–0.10) suggests instruction improvements may help; `none` (\<0.05) signals the agent's current instructions add no measurable lift over a vanilla agent — consider strengthening domain-specific antipatterns and re-running. Token cost is informational (logged in scores.json) but not part of the verdict — prioritize recall/F1 delta as the primary signal. - **A/B blind spot — role-specificity beyond recall**: for any agent whose domain is well-covered by general training data (structured rule application, documented conventions, standard code patterns), `none` AB verdict does NOT mean "retire the agent". Their specialization shows up in severity accuracy, output actionability, token efficiency, and scope discipline — not recall alone. The benchmark measures all four: `delta_severity_accuracy` (correct prioritization), `delta_format_score` (structured, actionable output), `token_ratio` (conciseness), and `scope_fp` (domain refusal). A `none` ΔRecall result paired with positive ΔSevAcc, ΔFmt, and negative ΔTokens still confirms the specialist earns its cost — use ΔSevAcc and ΔFmt as the primary evidence in this case. - **AB mode nesting**: Phase 2b spawns `general-purpose` baseline agents inside the pipeline subagent. Phase 3 spawns `general-purpose` scorer agents inside the same pipeline subagent. All at 2 levels (main → pipeline → agents) — no additional depth.