Tune Agent

/tune-agent — Reflexion patch for an agent's prompt

/tune-agent <name> spawns the agent-tuner agent with collected evidence and proposes a focused patch to that agent's .md file.

When to invoke

• /mine-transcripts produced agent-tuning.md listing agents with ≥2 failures
• /eval shows a regression for a specific agent vs prior runs
• Verifier or critic flagged the same failure pattern from the same agent twice in this session
• The user says "this agent keeps doing X — fix it"

What you do

1. Resolve the agent. ~/.claude/agents/<name>.md must exist. If not, stop.

2. Collect evidence. Pull from:

   • Most recent ~/.claude/telemetry/mining-runs/*/agent-tuning.md (if applicable)
   • Verifier/critic outputs in this session that mention the agent
   • Any context the user just provided

   Need ≥2 pieces. If you have only 1, tell the user — wait for more before tuning.

3. Spawn agent-tuner. Use the Agent tool with subagent_type: agent-tuner. Pass the evidence as a structured prompt:

   Agent to tune: <name>
   Evidence:
   1. <task summary> — <what went wrong> — session=<id>
   2. <task summary> — <what went wrong> — session=<id>
   3. <task summary> — <what went wrong> — session=<id>
   
   Desired behavior: <if user specified, otherwise infer>
   

4. Read the tuner's output. It ends with one of:

   • RESULT_agent-tuner=PATCHED — proceed to step 5 (A/B gate). Do NOT leave the patch live.
   • RESULT_agent-tuner=INSUFFICIENT_EVIDENCE — tell user, suggest waiting
   • RESULT_agent-tuner=AMBIGUOUS — surface options to user
   • RESULT_agent-tuner=NO_PATCH_NEEDED — evidence didn't add up; explain

5. Stage as candidate — NEVER apply directly. Reflexion-style prompt patching has no published validation (2026-07 deep-research pass found zero peer-reviewed evidence it works); the A/B gate IS the validation. Immediately after PATCHED:

   mkdir -p ~/.claude/rl/policy/<name>/candidates
   cp ~/.claude/agents/<name>.md ~/.claude/rl/policy/<name>/candidates/tuner-$(date +%Y-%m-%d).md
   cp "$(ls -t ~/.claude/agents-history/<name>-*.md | head -1)" ~/.claude/agents/<name>.md   # restore live prompt
   

   Surface the diff and rationale to the user, then log the staging to ~/.claude/agents-history/CHANGELOG.md.

6. A/B gate (mandatory before promotion). Run /ab-test <name>. Promote ONLY if the decision meets the promotion rule (candidate mean > current mean + max(stderr, 0.05), n ≥ 10):

   python3 ~/.claude/rl/ab_test.py <name> --runs 3
   # if decision.json recommends the candidate:
   python3 ~/.claude/rl/ab_test.py <name> --promote tuner-<date>
   

   If scenarios for the agent don't exist yet, run /eval-author <name> first — no scenarios means no gate means no promotion. An inconclusive A/B leaves the candidate staged; re-run after more scenarios accumulate rather than promoting on vibes.

7. Record a prediction (mandatory on promotion). Every promoted patch carries a falsifiable prediction the ledger scores later (AHE decision observability, arXiv 2604.25850):

   python3 ~/.claude/rl/prediction_ledger.py add \
     --change "tune-agent: <name> <one-line patch summary>" \
     --metric mean_reward --agent <name> --direction up --horizon-days 30
   

   Pick the metric the evidence actually implicates (mean_reward, correction_rate down, verifier_fail_rate down). /rl-status scores it when due — a wrong prediction is evidence for reverting.

8. Post-promotion re-eval. After a promotion, run /eval <name> to confirm no regression on behaviors the A/B scenarios didn't cover.

Cost note

The tuner runs on Opus (precise prompt engineering matters). One invocation is typically <$0.50. If you find yourself tuning the same agent more than weekly, the agent's design is wrong — propose a redesign, not another patch.

Anti-patterns to refuse

• Tuning on a single occurrence (refuse — collect more evidence first)
• Tuning based on user vibes rather than logged failures
• Tuning multiple agents in one shot — each /tune-agent is one agent
• Skipping the post-patch /eval — you must verify the patch doesn't regress