# Prompt Eval > Self-improvement framework for Claude Code prompts. Pass a profile name to evaluate variations of that target prompt via a 3-level cascade (stability via embeddings, decision-consistency via parsing, and pairwise quality via a judge bracket). The skill itself is the team lead — it dispatches all runner teammates directly at the top level (no nested teams), so every parallel agent is visible in the Claude Code Agent Teams tmux split. Use when this capability is needed. - Skill: `tomevault-io/prompt-eval` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/prompt-eval` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/prompt-eval/raw - Safety review: pending (external: skill-scanner PASS, skillspector CAUTION) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/prompt-eval --- # Activation Invoked as `/prompt-eval ` (filename without `.yml`, e.g. `ai-board.specify`). Optional flags: - `--mode auto` (overrides profile mode; requires `limits` set) - `--max-budget ` (overrides `limits.max_budget_usd`) - `--max-rounds ` (overrides `limits.max_rounds`) - `--runs ` (overrides `eval.runs_per_hypothesis` for ad-hoc cheaper passes) # You Are the Team Lead You orchestrate the entire run from this session. **All teammates you dispatch are at the top level** — runner teammates only. You never tell a teammate to dispatch another teammate. The `runner` agent (see `agents/runner.md`) is the only role you spawn. This is a deliberate flat architecture: - It avoids the nested-team limitation of Claude Code Agent Teams - It makes every parallel agent visible in the tmux split - It keeps each teammate's context tightly scoped to its single run # Phase 1 — Bootstrap ## 1.0 Pre-flight checks (fail fast, fail loudly) Before doing any work, verify the environment can run a full cascade. Otherwise the user pays for a partial run that silently downgrades. ```bash # Required for the bracket judge and the runners. command -v claude >/dev/null || { echo "ABORT: claude CLI not on PATH"; exit 1; } command -v bun >/dev/null || { echo "ABORT: bun not on PATH"; exit 1; } command -v patch >/dev/null || { echo "ABORT: patch not on PATH"; exit 1; } # Required for L1 stability (Mistral embeddings). If absent, L1 will be SKIPPED # and every hypothesis is treated as L1-passing — the bracket becomes the only # quality signal. WARN explicitly so the user can opt to abort and set the key. if [ -z "${MISTRAL_API_KEY:-}" ]; then echo "WARN: MISTRAL_API_KEY not set. L1 stability will be skipped; hypotheses are admitted to L2/bracket without per-pair embedding similarity. Continue? (Y/N)" fi ``` In `mode: auto`: if `MISTRAL_API_KEY` is absent, proceed with the warning recorded in the round report (no interactive prompt). In `mode: semi-auto`: ask the user explicitly before proceeding. ## 1.1 Resolve plugin_root `plugin_root` is the absolute path to this plugin's install directory. From within this skill: ```bash plugin_root="$(cd "$(dirname "$(realpath ./skills/prompt-eval/SKILL.md)")/../.." && pwd)" # or use $CLAUDE_PLUGIN_ROOT if Claude Code exposes it; fall back to the resolved path. ``` The CLI lives at `$plugin_root/scripts/prompt-eval`. The first profile lives at `$plugin_root/profiles/.yml`. ## 1.2 Resolve and load the profile Profiles can live in two locations. Resolve in this priority order: 1. **`$HOME/.prompt-eval/profiles/.yml`** — user profiles (created by `/prompt-eval-init`, persistent across plugin updates). 2. **`$plugin_root/profiles/.yml`** — built-in profiles shipped with the plugin (e.g. `ai-board.specify` as a starting example). ```bash profile_path="$HOME/.prompt-eval/profiles/.yml" [ -f "$profile_path" ] || profile_path="$plugin_root/profiles/.yml" [ -f "$profile_path" ] || { echo "Profile not found in either ~/.prompt-eval/profiles/ or $plugin_root/profiles/"; exit 1; } ``` A user profile with the same name as a built-in **overrides** the built-in — useful for forking the bundled `ai-board.specify` profile to your own variant without touching the plugin. Then load and validate: ```bash bun -e "import('$plugin_root/lib/profile-loader.ts').then(m => m.loadProfile('$profile_path')).then(p => console.log(JSON.stringify(p)))" ``` If `--mode auto` is in effect (after CLI override), require `limits.max_rounds > 0` AND `limits.max_budget_usd > 0`. Otherwise abort with: `"auto mode requires positive max_rounds and max_budget_usd in profile.limits"`. Apply CLI overrides to the in-memory profile object: `--max-budget` → `limits.max_budget_usd`, `--max-rounds` → `limits.max_rounds`, `--runs` → `eval.runs_per_hypothesis`. ## 1.3 Initialise run state ```bash run_id=$(bun -e "import('$plugin_root/lib/run-id.ts').then(m => console.log(m.generateRunId('')))") run_dir="$HOME/.prompt-eval/runs/$run_id" clones_root="$HOME/.prompt-eval/clones/$run_id" mkdir -p "$run_dir" "$clones_root" ``` Copy the original prompt to `$run_dir/original-baseline.md` (frozen reference): ```bash cp "/" "$run_dir/original-baseline.md" ``` Initialise `eval-run.yml`: ```bash bun -e "import('$plugin_root/lib/state.ts').then(m => m.initState('$run_dir', { run_id: '$run_id', profile_path: '', mode: '', baseline_path: '$run_dir/original-baseline.md' }))" ``` ## 1.4 Determine round 1 hypotheses Sourcing priority: ### (a) `profile.initial_hypotheses` non-empty Use them directly. Persist under `eval-run.yml.hypotheses_round_1` and proceed. ### (b) `mode: auto` AND `initial_hypotheses` empty — generate them yourself **Read `/references/prompt-best-practices.md` first.** It defines: - **9 best-practice axes** — 6 universal (1, 2, 3, 4, 5, 8: Clarity, Directness, Output Guidelines, Process Steps, Specificity, Robustness) and 3 surface-conditional (6, 7, 9: XML Structure, Examples, Parameter Tuning). Surface-conditional axes only apply when the prompt has the surface they target — see "Applying the axes" in the reference. Don't propose hypotheses for axes whose surface doesn't exist on this target. - **A "beyond axes" category** for domain-specific tweaks (cost/length, model swap, constraint removal, section reordering) - **Generation heuristics** you must follow (one axis per hypothesis, prefer additions, small diffs, cover multiple axes, prefix with `[Axis N: ]` or `[Beyond: ]`) - **Size-aware hypothesis design** — compute the per-hypothesis line budget from the target prompt's current size: ≤50 lines = up to +200%, 50-200 lines = ≤30%, ≥200 lines = ≤10%. Apply size-saving patterns from the reference (external example files + reference, inline one-liner rationale, terse fallbacks) when over budget, or split the change across rounds. Bloat regresses other axes — the bracket judge sees this. **Then look for an existing audit.** Before generating from scratch: ```bash target_basename="$(basename '' .md)" latest_audit="$(ls -1 $HOME/.prompt-eval/audits/${target_basename}-*.md 2>/dev/null | sort | tail -n1)" ``` If `$latest_audit` exists, parse its "## A/B test candidates" section. Each candidate already comes with a description, axis tag, and unified diff. **Use those as your starting set.** Only add self-generated hypotheses if the audit didn't cover an axis or category you think is worth probing this round. Surface this to the run state with provenance: `eval-run.yml.hypotheses_round_1[k].source = "audit:"` for audit-derived ones, `"generated"` for the rest. Then: 1. Read `$run_dir/original-baseline.md` (the target prompt) 2. Identify weaknesses by axis OR check the audit's findings 3. Propose 3–5 hypotheses combining audit candidates (priority) + axis-based additions for diversity 4. Generate / extract a unified diff for each 5. Persist as separate files (per the diff persistence rule below) with metadata + `source` in `eval-run.yml.hypotheses_round_1` 6. Proceed to Phase 2 without user intervention ### (c) `mode: semi-auto` AND `initial_hypotheses` empty — interactive loop with the user Even in semi-auto, **check `~/.prompt-eval/audits/` first**. If a recent audit exists for the target, mention it to the user up-front and offer its A/B candidates as a starting menu: > "I found a recent audit at `` with N A/B candidates. Want me to use them as the round-1 hypotheses? (Y/N) Or describe your own." This skips the user re-typing what the audit already produced. > "I'm preparing round 1 for ``. The target prompt is at `` (in ``). Describe your first hypothesis in plain language (e.g. 'tighten the AUTO security keyword bonus from +3 to +2'). I'll generate a unified diff and ask you to confirm before adding it." For each hypothesis: 1. User describes in plain language 2. You produce a unified diff against `$run_dir/original-baseline.md` and show it 3. User approves / edits / rejects 4. Repeat until 3-5 hypotheses are collected, or user says `go` Cap at `profile.eval.max_hypotheses_per_round`. **Persist diffs as separate files, NOT inline in the YAML state.** YAML serialisers fold long strings into `>` block scalars, which corrupts unified diffs (line-wrapping, lost leading whitespace, blank lines inserted) past `git apply`/`patch` recovery. For each approved hypothesis `Hn`: ```bash hk_dir="$run_dir/rounds/round-1/hypotheses/H$n" mkdir -p "$hk_dir" # Use printf with %s to preserve every byte verbatim (no echo -e quirks). printf '%s' "$DIFF_CONTENT" > "$hk_dir/variation.diff" printf '%s' "$DESCRIPTION" > "$hk_dir/description.md" ``` Then persist metadata only into `eval-run.yml.hypotheses_round_1`. Each entry follows this schema: ```yaml hypotheses_round_1: - id: H1 description: "(short label, single line)" diff_path: "rounds/round-1/hypotheses/H1/variation.diff" # relative to $run_dir source: "audit:" | "generated" # provenance for traceability - id: H2 ... ``` If you must hand-write YAML for this (rather than going through `lib/state.ts writeState`), use the `yaml` package with `{ lineWidth: 0 }` to disable folding. For a complete walked-through example of one round (hypotheses → runs → bracket → decision → report), see [`examples/sample-eval-round.md`](../../examples/sample-eval-round.md). # Phase 2 — Per Round (loop) For each round `N` starting at 1: ## 2.1 Prepare baseline clone ```bash round_dir="$run_dir/rounds/round-$N" mkdir -p "$round_dir" baseline_clone="$clones_root/round-$N-baseline" echo "{\"source\":\"\",\"dest\":\"$baseline_clone\"}" \ | "$plugin_root/scripts/prompt-eval" clone-shared # Replace the prompt file inside the clone with the round-N baseline content. cp "$run_dir/" "$baseline_clone/" echo "{\"repoPath\":\"$baseline_clone\",\"message\":\"snapshot round-$N baseline\"}" \ | "$plugin_root/scripts/prompt-eval" commit-all # Save baseline.md for the round audit trail. cp "$baseline_clone/" "$round_dir/baseline.md" ``` ## 2.2 Prepare hypothesis-base clones (one per hypothesis, in parallel) For each hypothesis `Hk` in this round, in parallel via background bash: ```bash hk_base="$clones_root/round-$N-$Hk-base" echo "{\"source\":\"$baseline_clone\",\"dest\":\"$hk_base\"}" \ | "$plugin_root/scripts/prompt-eval" clone-shared # Read the hypothesis diff from the file referenced in eval-run.yml.hypotheses_round_$N[Hk].diff_path. diff_path="$run_dir/$(yq '.hypotheses_round_'$N'[] | select(.id == "'$Hk'") | .diff_path' "$run_dir/eval-run.yml")" diff_content=$(cat "$diff_path") # JSON-escape the diff content for the CLI invocation. echo "{\"cwd\":\"$hk_base\",\"diff\":$(jq -Rs . <<< "$diff_content")}" \ | "$plugin_root/scripts/prompt-eval" apply-diff echo "{\"repoPath\":\"$hk_base\",\"message\":\"apply $Hk\"}" \ | "$plugin_root/scripts/prompt-eval" commit-all ``` If `apply-diff` fails for any `Hk`, mark that hypothesis as `rejected:patch_failed` in the round state and skip it from runner dispatch. ## 2.3 Prepare run clones (M = runs_per_hypothesis per hypothesis, in parallel) For each `(Hk, run_index k in 1..M)`: ```bash run_clone="$clones_root/round-$N-$Hk-run-$k" echo "{\"source\":\"$clones_root/round-$N-$Hk-base\",\"dest\":\"$run_clone\"}" \ | "$plugin_root/scripts/prompt-eval" clone-shared ``` ## 2.4 Dispatch ALL runner teammates in parallel This is the visibility-critical step. ### 2.4.0 — Create the team FIRST Before any `Agent` dispatch, call `TeamCreate` exactly once for this round to instantiate the team that all runners (and the baseline runners) will join. Without this, `Agent` calls referencing `team_name` spawn teammate placeholders with no team attached — they die immediately with `0 tool uses` and produce no work. (Symptom seen in practice: every runner reports "Done" with no output and no usage.) ``` TeamCreate({ name: "prompt-eval-round-", // additional fields per the TeamCreate schema (read it before invoking) }) ``` ### 2.4.1 — Then dispatch all runners Use the **Agent tool with multiple parallel tool calls in a single message**, one teammate per `(Hk, k)` — plus the baseline runs (you need 3 runs of the round-N baseline for the bracket judging to have a like-for-like comparison). For each `(Hk, k)` and each `(baseline, k)`: - `subagent_type`: `runner` - `name`: `runner--` (so they're addressable in tmux split) - `team_name`: `prompt-eval-round-` - `prompt`: a focused prompt that gives the runner exactly its inputs (see `agents/runner.md` for the contract, and `examples/sample-runner-output.md` for the response shape): - hypothesis_id, run_index - clone_path = `$run_clone` - invoke = `` - payload = `` - output_artifact = `` - outputs_root = `$round_dir/hypotheses/$Hk/outputs` - timeout_ms = `600000` (default — 10 min wall-time; covers most slash-commands. Profiles may override per target.) **Concurrency cap**: dispatch at most `eval.concurrency_per_hypothesis × number_of_qualified_hypotheses` runners simultaneously, but never more than `15` runners total. (Rationale: Claude Code Agent Teams hard cap is 16 members per team — leave 1 slot for the lead itself.) If your N×M exceeds `15`, dispatch in waves: first 15, await, then the remainder. Wait for all runner teammates to return. Each returns a JSON status (see `agents/runner.md` §Step 8 and the canonical example at `examples/sample-runner-output.md`). Persist these into `$round_dir/hypotheses/$Hk/eval/runs.json`. **Resilience to lost SendMessage.** Teammate→lead `SendMessage` is best-effort: in practice some runners' messages don't reach the lead even though the runner did its work. The runner contract therefore requires every runner to write its report to `/run-.report.json` BEFORE calling SendMessage. After the dispatch wave returns, **for every runner whose SendMessage didn't arrive, read its `report.json` from disk** instead. Only treat a runner as truly missing if both the SendMessage and the on-disk report are absent. After all runners return, sum every runner's `usage.cost_usd` into a per-round running total `runner_cost_round_$N`. Also increment `state.budget_consumed_usd` by the same sum via `lib/state.ts addBudget`. **Check budget gate**: if exceeded, finalise the round as-is (no more dispatches in subsequent rounds). In stream-json mode, the runner's cost is `total_cost_usd` on the final `"type":"result"` event — see `agents/runner.md` §Step 7. `usage.cost_usd` does not exist in stream-json and parses to 0. ## 2.5 Aggregate L1 + L2 per hypothesis For each hypothesis `Hk`: If `≥3` of M runs returned a non-`ok` status, mark `Hk` as `rejected:unreliable`. Skip L1/L2. (Rationale for the `3` threshold: with the default `M=3`, this means "the entire batch failed" — no signal possible. With higher M, a 3-failure floor still preserves enough surviving runs for a meaningful L1.) Otherwise: ```bash # L1 echo "{\"runOutputs\":[],\"embedding_model\":\"\",\"threshold\":}" \ | "$plugin_root/scripts/prompt-eval" score-l1 > "$round_dir/hypotheses/$Hk/eval/l1.json" ``` Read `gate` from l1.json. If `fail` → `rejected:unstable`, skip L2. ```bash # L2 (only if profile.eval.level2_decisions.skip != true) echo "{\"runOutputs\":[...],\"parser\":\"\",\"sectionName\":\"
\",\"decisionKey\":\"\",\"thresholdPct\":}" \ | "$plugin_root/scripts/prompt-eval" score-l2 > "$round_dir/hypotheses/$Hk/eval/l2.json" ``` If `gate==fail` → `rejected:inconsistent`. Else → `qualified`. Persist `$round_dir/hypotheses/$Hk/eval/status.json` with the final classification, l1, l2, total_usd for that hypothesis. ## 2.6 Pairwise bracket on qualified survivors If 0 qualified survivors → `decision: rollback`, skip bracket. Otherwise: For each side (baseline + each qualified hypothesis), pick the **centroid run** — the run with the median pairwise similarity to its peers (read from L1's `pair_similarities`). The baseline must therefore be run `runs_per_hypothesis` times in Phase 2.4 alongside the hypotheses (apples-to-apples comparison; using a single baseline-snapshot vs. M-aggregated hypothesis runs would bias the bracket). Build participants list `[baseline-centroid, ...qualified-centroids-in-order]`. Single-elimination bracket: pair adjacent participants, judge them, winner advances. For each match `(a, b)`: ```bash echo "{\"rubric\":\"\",\"specA\":\"\",\"specB\":\"\",\"judge_model\":\"\",\"double_blind\":}" \ | "$plugin_root/scripts/prompt-eval" judge ``` Returns `{"verdict":"A"|"B"|"tied"}`. Tied resolves in favour of the participant earlier in the list (baseline is always at index 0, so tied always favours baseline). Sum the judge's cost into a per-round running total `judge_cost_round_$N`. Also `addBudget` it to `state.budget_consumed_usd`. Persist `$round_dir/bracket.json` with each match record (verdict + cost), so the report can show per-match breakdown if needed later. ## 2.7 Decide - If bracket winner is `baseline` → **rollback**. `state.baseline_path` unchanged. - Else → **adopt**. Update `state.baseline_path` to point at the winning hypothesis's variation file (`$round_dir/hypotheses//variation.md` — write that file from the hypothesis-base clone). Render `$round_dir/round-report.md` using the report renderer. Pass `runner_cost_usd` and `judge_cost_usd` (the per-round running totals tracked above) so the report shows the breakdown: ```bash bun -e "import('$plugin_root/lib/report.ts').then(m => process.stdout.write(m.renderRoundReport({ ..., total_usd: , runner_cost_usd: , judge_cost_usd: })))" > "$round_dir/round-report.md" ``` Write `$round_dir/decision.json` capturing the round's outcome. ## 2.8 Cleanup clones AND the team for this round Clones: ```bash echo "{\"path\":\"$clones_root/round-$N-baseline\"}" | "$plugin_root/scripts/prompt-eval" remove-clone for hk in $hypotheses; do echo "{\"path\":\"$clones_root/round-$N-$hk-base\"}" | "$plugin_root/scripts/prompt-eval" remove-clone for k in $(seq 1 $M); do echo "{\"path\":\"$clones_root/round-$N-$hk-run-$k\"}" | "$plugin_root/scripts/prompt-eval" remove-clone done done ``` Team (the one we created in Step 2.4.0): ``` TeamDelete({ name: "prompt-eval-round-" }) ``` (Without the delete, dangling teams accumulate across rounds. Symptom: TeamCreate of the next round fails with "team already exists".) `$run_dir` (state, outputs, reports) is preserved. ## 2.9 Bump round counter ```bash bun -e "import('$plugin_root/lib/state.ts').then(m => m.bumpRound('$run_dir'))" ``` ## 2.10 Stop criteria check In order, the first that fires wins: 1. **Convergence** — if the last `2` rounds both ended in `rollback`. (Rationale: a single rollback can be noise; two in a row is a clear plateau signal.) 2. **Budget** — if `state.budget_consumed_usd >= profile.limits.max_budget_usd` 3. **Round cap** — if `state.rounds_completed >= profile.limits.max_rounds` 4. **User stop** (semi-auto only) — explicit user "stop" at the round checkpoint If any fires: ```bash bun -e "import('$plugin_root/lib/report.ts').then(m => process.stdout.write(m.renderRoundReport(...))) " > "$run_dir/final-report.md" # (or a separate final-report renderer if you have one) ``` Present the final report path to the user and return. ## 2.11 Round checkpoint and next-round hypotheses Whatever the mode, the next-round hypotheses MUST be grounded in `/references/prompt-best-practices.md`. Same rules as Phase 1.4 (b): one axis per hypothesis, additions preferred, small diffs, cover multiple axes, prefix descriptions with `[Axis N: ]`. When proposing the next round, read the round report first and pick axes that **haven't been explored yet** (or that were explored but the hypothesis was rejected for a fixable reason — e.g. unstable because the diff was too large; retry with a tighter version). ### `mode: semi-auto` Before dispatching round N+1: 1. Print `$round_dir/round-report.md` to the user 2. Propose 3-5 new hypotheses (one per axis, see best-practices) 3. Ask the user to approve / edit / drop / add. Wait for confirmation 4. Persist into `eval-run.yml.hypotheses_round_$((N+1))` 5. Loop ### `mode: auto` Skip the checkpoint. Propose 3-5 new hypotheses (still one per axis), record their description + diff_path in `eval-run.yml.hypotheses_round_$((N+1))`, and proceed. # Notes - **All teammate dispatches are PARALLEL.** Use multiple Agent tool calls in a single message — that's how Claude Code parallelises them and renders the tmux split. - **All paths to teammates must be absolute.** - **Persist after every important step.** Crashes mid-round must leave a resumable state (use `lib/state.ts` `writeState`). - The CLI subcommands are documented in `lib/cli.ts`. Every CLI invocation reads JSON from stdin and writes JSON to stdout. - If the user sends Ctrl-C: stop dispatching, but let in-flight runners complete. Persist the partial state. The user can resume from `eval-run.yml` later (resume CLI is post-MVP). --- > Source: [bfernandez31/prompt-eval](https://github.com/bfernandez31/prompt-eval) — distributed by [TomeVault](https://tomevault.io).