# Prompt Optimizer > prompt-optimizer - Skill: `masharratt/prompt-optimizer` (Agent Skill, multi-file: 17 files) - Install (CLI): `npx skillmds@latest add masharratt/prompt-optimizer` - Raw SKILL.md: https://api.skillmd.com/api/skills/masharratt/prompt-optimizer/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: masharratt (https://skillmd.com/u/masharratt) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/masharratt/prompt-optimizer --- # prompt-optimizer Shared prompt-optimization engine. Hill-climbs a prompt template against a project-supplied rubric: mutate → score on held-out-safe fixtures → accept improvements → validate the winner never saw during training. Generic across projects; each project supplies its own **plugin** (target + rubric + fixtures). This skill dir (`engine/`) is symlinked into every project via `~/.claude/skills` — it is READ-ONLY code. Nothing it runs ever writes here. ## Why a shared engine Multiple projects independently forked a one-off optimizer and hand-rolled the same fixes (held-out validation, tri-state no-run handling, deterministic scoring, reject-and-regenerate, cost-aware tie-breaks). This skill is the one place those fixes live, so every consumer inherits them for free instead of re-solving the same bugs per project. ## Architecture ``` engine/ types.ts the plugin contract (Target, Rubric, Fixture, RubricScore, Hit) paths.ts project-local path resolver (BLOCKER-1 — see below) rubric-core.ts generic aggregate + isImprovement + cost-Pareto tie-break budget.ts BudgetTracker + per-model pricing table source-patcher.ts sentinel-delimited source patcher (optional, plugin-invoked) eval.ts eval loop: tri-state exclusion, temp-0 scoring, reject-and-regenerate mutator.ts rubric-agnostic strategy-rotation mutator optimize.ts orchestrator: baseline -> mutate/accept on train -> holdout gate -> report execute.sh entry point: resolves the project-local plugin, runs optimize.ts ``` ## The plugin contract A project supplies a **plugin**: real TS/JS modules (not declarative config), because a real rubric needs logic (e.g. computed-age/scene scoring can't be expressed as JSON). ```ts interface Target { id: string; loadTemplate(): string | Promise; renderPrompt(template: string, fixture: Fixture): { prompt: string }; // The ONLY place a provider SDK may be imported — never inside engine/*. generate(prompt: string, options: { temperature: number }): Promise; extractScript(raw: string): { ok: true; text: string } | { ok: false; reason: string }; // Optional — declare only if the target needs it. Both default to // engine behavior when absent, so every existing plugin keeps working. evalTemperature?: number; // lowest temp this target can honor (default: 0) nondeterministic?: boolean; // provider ignores temperature 0 (default: false) pricing?: { input: number; output: number }; // USD/1M tokens (default: engine's built-in table) // Optional, all three together: opts into `--apply` source patching (see // "Source-patch auto-apply" below). Omit all three (default) and this // target keeps writing only to templates/.md, unchanged. sourceFile?: string; // path to the prompt's source, relative to the CONSUMING project's cwd varMap?: Record; // {{PLACEHOLDER}} -> local expression string, e.g. { NAME: 'input.name' } assignmentVar?: string; // local var name for the emitted `const = \`...\`;` } interface Rubric { categories: string[]; describe(): string; // fed to the mutator — no engine-owned rubric text // E1: may return RubricScore directly OR Promise. The engine // always awaits it, so an async rubric (LLM-as-judge, network lookup) // works unchanged next to every existing synchronous rubric. Known limit: // an async rubric adds one call per scored example and its own cost. // budget.ts only tracks Target.generate costs, never a rubric's own spend. score(text: string, ctx: Fixture): RubricScore | Promise; regenerateOn?: string[]; // categories that trigger a bounded 1-retry regen } interface Fixture { id: string; split: 'train' | 'holdout'; // frozen — the holdout gate can never train on these [key: string]: unknown; // whatever data your renderPrompt/score need } ``` Notice `mutateTemplate` calls the LLM only through `target.generate` — the engine reuses the plugin's own provider client for the meta-prompt that asks the model to rewrite the template. The engine itself never imports a provider SDK (BLOCKER-2): `grep -rn "from 'openai'" engine/` must return zero, and it does. ## Project-local plugin convention ``` /.claude/prompt-optimizer/ config.json { "": { "target": "./targets/x.ts", "rubric": "./rubrics/x.ts", "fixtures": "./fixtures/x.json" } } targets/.ts implements Target rubrics/.ts implements Rubric fixtures/.json Fixture[] with frozen split tags # all writable below here, created on demand — never under the engine's own dir state/.json per-iteration log runs/-.md human-readable run report templates/.md the current winning template backups/-.md seed backed up here before EVERY overwrite of templates/.md (never lost — see fix #1 below), plus source-patcher backups if a plugin invokes it _budget.json spend ledger. Per-project, SHARED across every target, and persists across every run forever. --budget caps only the current run's fresh spend; --lifetime-budget (optional) caps this ledger's cumulative total across all runs. ``` Dynamic-imports the `.ts`/`.js` modules from the project's own cwd via `import()` (works with plain JS or, if run through `npx tsx`, `.ts` directly). **Absent `config.json`** → the engine prints `no plugin configured` and exits 0. Inheriting this shared skill never breaks a project that hasn't opted in. ## BLOCKER-1 — state isolation Every writable path resolves under `/.claude/prompt-optimizer/` (`engine/paths.ts`), never under the engine's own `SKILL_DIR`. Two projects running this same symlinked engine write to two disjoint directories — concurrency-safe by construction, no shared budget ledger, no cross-project fixture leakage. ## BLOCKER-2 — provider-agnostic engine `engine/*` imports only `p-limit` and node builtins. A provider client (OpenAI SDK, xAI, Gemini, whatever) lives ENTIRELY inside a plugin's `Target.generate`. Peer dependencies each consumer project needs: - `tsx` — to run `.ts` plugin modules directly (or ship plugins as plain `.js`) - its own provider SDK (`openai`, `@google/genai`, etc.) — inside the plugin only - `p-limit` — eval concurrency (already a dependency here) - `vitest` — to run the engine's own test suite in that project, if desired ## Engine fixes baked in 1. **Held-out split** (`optimize.ts`) — mutate/accept only on `split:'train'`; score the winner ONCE (or, for a nondeterministic target, `--holdout-repeats` times — see fix #3 below) on `split:'holdout'` after convergence. If holdout regresses relative to the baseline's holdout score, the run is labeled **OVERFIT** and the win is refused — the reported/persisted template reverts to the baseline. Before ANY overwrite of `templates/.md`, the prior template is backed up to `backups/-.md` and a unified diff (seed vs. final) is appended to the run report — the human-authored seed is never destroyed. 2. **Tri-state no-run exclusion** (`eval.ts`, `rubric-core.ts`) — an `extractScript` `ok:false` result (parse failure / refusal / empty) is excluded from the aggregate and counted separately. The eval aborts if fewer than 50% of fixtures produced a scoreable result. 3. **Temperature-0 scoring, with an opt-out** (`eval.ts`) — eval pins `temperature: 0` for a stable ranking by default. A target that cannot honor temperature 0 (e.g. a provider that rejects any temperature but 1) declares `Target.evalTemperature` to override this; the engine then stamps a **NONDETERMINISTIC SCORING** warning into the run report and per-iteration state, since every total in that run is noisy rather than a clean deterministic measurement. **A provider may accept temperature 0 and ignore it.** Measured true for xAI Grok: two runs of one byte-identical prompt set gave train baselines of 3, 8 and 5, and a direct two-call probe returned 175 vs 201 words diverging at word 5. Nondeterminism is a property of the provider, not of the number we send it, so such a target declares `nondeterministic: true` and gets the same warning and repeat sampling. Without it the riskiest case had the LEAST protection, silently reporting noise as an exact measurement. Declaring `nondeterministic: false` cannot suppress a non-zero `evalTemperature` — a target may not opt out of noise it demonstrably generates. Repeat sampling covers the HOLDOUT gate only. Train comparisons stay single-sample, so on a nondeterministic target an accepted train win reads honestly as "did not regress", not "improved". For a nondeterministic target, the holdout gate (fix #1) re-scores baseline AND final `--holdout-repeats` times (default 2) and requires the final to beat the baseline on EVERY repeat; mixed results (wins some, loses others) are labeled **INCONCLUSIVE** and the win is refused, same as OVERFIT — a noise floor that catches the case where a single lucky/unlucky sample would otherwise accept or reject a win on chance alone. Deterministic targets keep the original single-sample holdout check, no extra cost. 4. **Reject-and-regenerate** (`eval.ts`) — a rubric hit in a `regenerateOn` category triggers exactly ONE regeneration with a plugin-supplied corrective nudge appended to the prompt, then scores the retry result as-is (never a second retry). 5. **Cost-Pareto tie-break + sample-count integrity** (`rubric-core.ts`) — `isImprovement` requires no per-category regression; on an exact tie, prefers the candidate with fewer prompt tokens. A pure tie with no cost info is not accepted. Every comparison is a **per-example rate** (`total / ranCount`), never a raw sum, and a candidate that ran on FEWER examples than the incumbent is refused outright. An excluded example contributes 0 to the sum, so comparing raw sums let a template win by breaking its own output. Found live (L9): a candidate scoring `total=2 ran=11 excluded=4` was accepted over a `total=3 ran=15 excluded=0` baseline, having improved nothing except how many generations failed extraction. Rate normalisation alone does not close this — 2/11 beats 3/15 — because the examples an exclusion removes are the hard ones. The ran-count floor is the load-bearing guard. The floor is deliberately strict: a real win that incidentally loses one example is refused too. Nothing distinguishes "dropped a hard example" from "dropped one at random", and refusing a real win costs one iteration while accepting a fake one corrupts the template that ships. 6. **Non-throwing, overridable pricing** (`budget.ts`) — `costFor` never throws on an unknown model: a `Target.pricing` override (USD per 1M tokens) is preferred when supplied, else the engine's built-in table; an unknown model with neither warns loudly and records cost as `$0` rather than crashing the shared budget ledger or silently misreporting spend. 7. **Rubric saturation note** (`optimize.ts`) — when the accepted/final train total hits exactly `0`, the report notes **RUBRIC SATURATED (total=0, no remaining signal)** — distinct from normal convergence, since a zero total means the rubric has no more violations left to find. 8. **Per-run budget, with an optional lifetime cap** (`budget.ts`, `optimize.ts`): `_budget.json` persists `spentUsd` FOREVER, shared across every target in the project. `--budget=N` caps only THIS run (`BudgetTracker.runSpent`, which always starts at `0`); it is never compared against the ledger's cumulative lifetime total. An OPTIONAL `--lifetime-budget=N` sets an absolute ceiling across every run that ever wrote to the ledger. `BudgetTracker.exhausted()` trips on either cap; `trippedCap()` reports which one (`'run' | 'lifetime' | null`) so every print/abort message states which cap tripped and shows both numbers, e.g. `[abort] run budget exhausted ($0.4501 of $0.45 this run; $1.0528 lifetime)`. The run report and per-iteration state record carry both `runSpendUsd` and the lifetime `spendUsd`, never only the lifetime figure. 9. **Async rubric scoring** (`types.ts`, `eval.ts`): `Rubric.score` may return `RubricScore` or `Promise`; the engine always `await`s it, so an LLM-as-judge or network-backed rubric works unchanged next to every existing synchronous rubric. Known limit: an async rubric's own call cost is NOT tracked by the budget ledger (only `Target.generate` costs are recorded there). 10. **Source-patch auto-apply, opt-in via `--apply`** (`optimize.ts`, `source-patcher.ts`, `types.ts`): DEFAULT OFF. A target that declares `sourceFile` + `varMap` + `assignmentVar` together (see the plugin contract above) can be patched directly: the engine replaces the region between `// PROMPT-OPTIMIZER:START id=` and `// PROMPT-OPTIMIZER:END` in that file with the winning template. The patch is attempted ONLY when ALL of these hold, so a refused or unchanged result can never reach a real source file: - `--apply` was passed, AND - the target declares `sourceFile` (with `varMap`/`assignmentVar`), AND - the run was NOT `--dry-run`, AND - the holdout gate produced a REAL win: not OVERFIT, not HOLDOUT INCONCLUSIVE, and the final template actually differs from the baseline template. Before writing, the replaced region is backed up to `backups/-.txt` (a distinct filename/extension from the `.md` template backups already written there). The apply step never throws: a missing file, a missing or malformed sentinel pair, or any other patcher error is caught, reported as a warning line, and recorded on the run's `applyResult` field and in the run report, without failing the run or losing the report. Every existing plugin (no `sourceFile` declared, or `--apply` never passed) is completely unaffected. ## Usage ```bash ./.claude/skills/prompt-optimizer/execute.sh \ [--dry-run] [--apply] [--budget=N] [--lifetime-budget=N] [--max-iters=N] [--patience=N] [--holdout-repeats=N] ``` Run from inside the consuming project (or `execute.sh` `cd`s there via its own resolution — see `execute.sh`). Resolves `/.claude/prompt-optimizer/config.json`, loads the named target's plugin, and runs `engine/optimize.ts` via `npx tsx`. `--budget=N` (default 5.0) caps THIS run only. `_budget.json` is a single ledger, per-project, SHARED across every target in that project, and its `spentUsd` persists across every run forever. `--budget` is compared only against this run's own fresh spend, never against that cumulative total. `--lifetime-budget=N` (default: unset, no lifetime ceiling) is an OPTIONAL absolute cap across every run that ever wrote to the ledger. Use this when you want a hard stop on total project spend in addition to the per-run cap. `--holdout-repeats=N` (default 2) — only takes effect for a target the engine considers nondeterministic: `evalTemperature` is not 0, OR the target declares `nondeterministic: true` (see fix #3 and L10). Ignored (no extra API calls) for deterministic targets. `--apply` (default OFF): see fix #10 above. Only patches a target's declared `sourceFile` when the run produced a real, non-refused win and the run was not `--dry-run`. Every target without a declared `sourceFile` is unaffected regardless of this flag. ## Testing Engine tests live at `engine/*.test.ts`, run with vitest, provider fully mocked (no live API calls, no `openai` import anywhere in engine code): ```bash npx vitest run .claude/skills/prompt-optimizer/engine --reporter=verbose ```