Prompt Optimizer

prompt-optimizer

Shared prompt-optimization engine. Hill-climbs a prompt template against a project-supplied rubric: mutate → score on held-out-safe fixtures → accept improvements → validate the winner never saw during training. Generic across projects; each project supplies its own plugin (target + rubric + fixtures). This skill dir (engine/) is symlinked into every project via ~/.claude/skills — it is READ-ONLY code. Nothing it runs ever writes here.

Why a shared engine

Multiple projects independently forked a one-off optimizer and hand-rolled the same fixes (held-out validation, tri-state no-run handling, deterministic scoring, reject-and-regenerate, cost-aware tie-breaks). This skill is the one place those fixes live, so every consumer inherits them for free instead of re-solving the same bugs per project.

Architecture

engine/
  types.ts          the plugin contract (Target, Rubric, Fixture, RubricScore, Hit)
  paths.ts           project-local path resolver (BLOCKER-1 — see below)
  rubric-core.ts     generic aggregate + isImprovement + cost-Pareto tie-break
  budget.ts          BudgetTracker + per-model pricing table
  source-patcher.ts  sentinel-delimited source patcher (optional, plugin-invoked)
  eval.ts            eval loop: tri-state exclusion, temp-0 scoring, reject-and-regenerate
  mutator.ts         rubric-agnostic strategy-rotation mutator
  optimize.ts        orchestrator: baseline -> mutate/accept on train -> holdout gate -> report
execute.sh           entry point: resolves the project-local plugin, runs optimize.ts

The plugin contract

A project supplies a plugin: real TS/JS modules (not declarative config), because a real rubric needs logic (e.g. computed-age/scene scoring can't be expressed as JSON).

interface Target {
  id: string;
  loadTemplate(): string | Promise<string>;
  renderPrompt(template: string, fixture: Fixture): { prompt: string };
  // The ONLY place a provider SDK may be imported — never inside engine/*.
  generate(prompt: string, options: { temperature: number }): Promise<GenerateResult>;
  extractScript(raw: string): { ok: true; text: string } | { ok: false; reason: string };
  // Optional — declare only if the target needs it. Both default to
  // engine behavior when absent, so every existing plugin keeps working.
  evalTemperature?: number;      // lowest temp this target can honor (default: 0)
  nondeterministic?: boolean;    // provider ignores temperature 0 (default: false)
  pricing?: { input: number; output: number }; // USD/1M tokens (default: engine's built-in table)
  // Optional, all three together: opts into `--apply` source patching (see
  // "Source-patch auto-apply" below). Omit all three (default) and this
  // target keeps writing only to templates/<id>.md, unchanged.
  sourceFile?: string;           // path to the prompt's source, relative to the CONSUMING project's cwd
  varMap?: Record<string, string>; // {{PLACEHOLDER}} -> local expression string, e.g. { NAME: 'input.name' }
  assignmentVar?: string;        // local var name for the emitted `const <assignmentVar> = \`...\`;`
}

interface Rubric {
  categories: string[];
  describe(): string;            // fed to the mutator — no engine-owned rubric text
  // E1: may return RubricScore directly OR Promise<RubricScore>. The engine
  // always awaits it, so an async rubric (LLM-as-judge, network lookup)
  // works unchanged next to every existing synchronous rubric. Known limit:
  // an async rubric adds one call per scored example and its own cost.
  // budget.ts only tracks Target.generate costs, never a rubric's own spend.
  score(text: string, ctx: Fixture): RubricScore | Promise<RubricScore>;
  regenerateOn?: string[];       // categories that trigger a bounded 1-retry regen
}

interface Fixture {
  id: string;
  split: 'train' | 'holdout';    // frozen — the holdout gate can never train on these
  [key: string]: unknown;        // whatever data your renderPrompt/score need
}

Notice mutateTemplate calls the LLM only through target.generate — the engine reuses the plugin's own provider client for the meta-prompt that asks the model to rewrite the template. The engine itself never imports a provider SDK (BLOCKER-2): grep -rn "from 'openai'" engine/ must return zero, and it does.

Project-local plugin convention

<project>/.claude/prompt-optimizer/
  config.json          { "<target-id>": { "target": "./targets/x.ts",
                                            "rubric": "./rubrics/x.ts",
                                            "fixtures": "./fixtures/x.json" } }
  targets/<id>.ts       implements Target
  rubrics/<id>.ts       implements Rubric
  fixtures/<id>.json    Fixture[] with frozen split tags
  # all writable below here, created on demand — never under the engine's own dir
  state/<id>.json       per-iteration log
  runs/<id>-<ts>.md     human-readable run report
  templates/<id>.md     the current winning template
  backups/<id>-<ts>.md  seed backed up here before EVERY overwrite of templates/<id>.md
                        (never lost — see fix #1 below), plus source-patcher
                        backups if a plugin invokes it
  _budget.json          spend ledger. Per-project, SHARED across every
                        target, and persists across every run forever.
                        --budget caps only the current run's fresh spend;
                        --lifetime-budget (optional) caps this ledger's
                        cumulative total across all runs.

Dynamic-imports the .ts/.js modules from the project's own cwd via import() (works with plain JS or, if run through npx tsx, .ts directly).

Absent config.json → the engine prints no plugin configured and exits 0. Inheriting this shared skill never breaks a project that hasn't opted in.

BLOCKER-1 — state isolation

Every writable path resolves under <cwd>/.claude/prompt-optimizer/ (engine/paths.ts), never under the engine's own SKILL_DIR. Two projects running this same symlinked engine write to two disjoint directories — concurrency-safe by construction, no shared budget ledger, no cross-project fixture leakage.

BLOCKER-2 — provider-agnostic engine

engine/* imports only p-limit and node builtins. A provider client (OpenAI SDK, xAI, Gemini, whatever) lives ENTIRELY inside a plugin's Target.generate. Peer dependencies each consumer project needs:

• tsx — to run .ts plugin modules directly (or ship plugins as plain .js)
• its own provider SDK (openai, @google/genai, etc.) — inside the plugin only
• p-limit — eval concurrency (already a dependency here)
• vitest — to run the engine's own test suite in that project, if desired

Engine fixes baked in

1. Held-out split (optimize.ts) — mutate/accept only on split:'train'; score the winner ONCE (or, for a nondeterministic target, --holdout-repeats times — see fix #3 below) on split:'holdout' after convergence. If holdout regresses relative to the baseline's holdout score, the run is labeled OVERFIT and the win is refused — the reported/persisted template reverts to the baseline. Before ANY overwrite of templates/<id>.md, the prior template is backed up to backups/<id>-<ISO-timestamp>.md and a unified diff (seed vs. final) is appended to the run report — the human-authored seed is never destroyed.

2. Tri-state no-run exclusion (eval.ts, rubric-core.ts) — an extractScript ok:false result (parse failure / refusal / empty) is excluded from the aggregate and counted separately. The eval aborts if fewer than 50% of fixtures produced a scoreable result.

3. Temperature-0 scoring, with an opt-out (eval.ts) — eval pins temperature: 0 for a stable ranking by default. A target that cannot honor temperature 0 (e.g. a provider that rejects any temperature but 1) declares Target.evalTemperature to override this; the engine then stamps a NONDETERMINISTIC SCORING warning into the run report and per-iteration state, since every total in that run is noisy rather than a clean deterministic measurement.

   A provider may accept temperature 0 and ignore it. Measured true for xAI Grok: two runs of one byte-identical prompt set gave train baselines of 3, 8 and 5, and a direct two-call probe returned 175 vs 201 words diverging at word 5. Nondeterminism is a property of the provider, not of the number we send it, so such a target declares nondeterministic: true and gets the same warning and repeat sampling. Without it the riskiest case had the LEAST protection, silently reporting noise as an exact measurement. Declaring nondeterministic: false cannot suppress a non-zero evalTemperature — a target may not opt out of noise it demonstrably generates.

   Repeat sampling covers the HOLDOUT gate only. Train comparisons stay single-sample, so on a nondeterministic target an accepted train win reads honestly as "did not regress", not "improved".

   For a nondeterministic target, the holdout gate (fix #1) re-scores baseline AND final --holdout-repeats times (default 2) and requires the final to beat the baseline on EVERY repeat; mixed results (wins some, loses others) are labeled INCONCLUSIVE and the win is refused, same as OVERFIT — a noise floor that catches the case where a single lucky/unlucky sample would otherwise accept or reject a win on chance alone. Deterministic targets keep the original single-sample holdout check, no extra cost.

4. Reject-and-regenerate (eval.ts) — a rubric hit in a regenerateOn category triggers exactly ONE regeneration with a plugin-supplied corrective nudge appended to the prompt, then scores the retry result as-is (never a second retry).

5. Cost-Pareto tie-break + sample-count integrity (rubric-core.ts) — isImprovement requires no per-category regression; on an exact tie, prefers the candidate with fewer prompt tokens. A pure tie with no cost info is not accepted.

   Every comparison is a per-example rate (total / ranCount), never a raw sum, and a candidate that ran on FEWER examples than the incumbent is refused outright. An excluded example contributes 0 to the sum, so comparing raw sums let a template win by breaking its own output. Found live (L9): a candidate scoring total=2 ran=11 excluded=4 was accepted over a total=3 ran=15 excluded=0 baseline, having improved nothing except how many generations failed extraction. Rate normalisation alone does not close this — 2/11 beats 3/15 — because the examples an exclusion removes are the hard ones. The ran-count floor is the load-bearing guard.

   The floor is deliberately strict: a real win that incidentally loses one example is refused too. Nothing distinguishes "dropped a hard example" from "dropped one at random", and refusing a real win costs one iteration while accepting a fake one corrupts the template that ships.

6. Non-throwing, overridable pricing (budget.ts) — costFor never throws on an unknown model: a Target.pricing override (USD per 1M tokens) is preferred when supplied, else the engine's built-in table; an unknown model with neither warns loudly and records cost as $0 rather than crashing the shared budget ledger or silently misreporting spend.

7. Rubric saturation note (optimize.ts) — when the accepted/final train total hits exactly 0, the report notes RUBRIC SATURATED (total=0, no remaining signal) — distinct from normal convergence, since a zero total means the rubric has no more violations left to find.

8. Per-run budget, with an optional lifetime cap (budget.ts, optimize.ts): _budget.json persists spentUsd FOREVER, shared across every target in the project. --budget=N caps only THIS run (BudgetTracker.runSpent, which always starts at 0); it is never compared against the ledger's cumulative lifetime total. An OPTIONAL --lifetime-budget=N sets an absolute ceiling across every run that ever wrote to the ledger. BudgetTracker.exhausted() trips on either cap; trippedCap() reports which one ('run' | 'lifetime' | null) so every print/abort message states which cap tripped and shows both numbers, e.g. [abort] run budget exhausted ($0.4501 of $0.45 this run; $1.0528 lifetime). The run report and per-iteration state record carry both runSpendUsd and the lifetime spendUsd, never only the lifetime figure.

9. Async rubric scoring (types.ts, eval.ts): Rubric.score may return RubricScore or Promise<RubricScore>; the engine always awaits it, so an LLM-as-judge or network-backed rubric works unchanged next to every existing synchronous rubric. Known limit: an async rubric's own call cost is NOT tracked by the budget ledger (only Target.generate costs are recorded there).

10. Source-patch auto-apply, opt-in via --apply (optimize.ts, source-patcher.ts, types.ts): DEFAULT OFF. A target that declares sourceFile + varMap + assignmentVar together (see the plugin contract above) can be patched directly: the engine replaces the region between // PROMPT-OPTIMIZER:START id=<target-id> and // PROMPT-OPTIMIZER:END in that file with the winning template. The patch is attempted ONLY when ALL of these hold, so a refused or unchanged result can never reach a real source file:

    • --apply was passed, AND
    • the target declares sourceFile (with varMap/assignmentVar), AND
    • the run was NOT --dry-run, AND
    • the holdout gate produced a REAL win: not OVERFIT, not HOLDOUT INCONCLUSIVE, and the final template actually differs from the baseline template. Before writing, the replaced region is backed up to backups/<id>-<ISO-timestamp>.txt (a distinct filename/extension from the .md template backups already written there). The apply step never throws: a missing file, a missing or malformed sentinel pair, or any other patcher error is caught, reported as a warning line, and recorded on the run's applyResult field and in the run report, without failing the run or losing the report. Every existing plugin (no sourceFile declared, or --apply never passed) is completely unaffected.

Usage

./.claude/skills/prompt-optimizer/execute.sh <target-id> \
  [--dry-run] [--apply] [--budget=N] [--lifetime-budget=N] [--max-iters=N] [--patience=N] [--holdout-repeats=N]

Run from inside the consuming project (or execute.sh cds there via its own resolution — see execute.sh). Resolves <project>/.claude/prompt-optimizer/config.json, loads the named target's plugin, and runs engine/optimize.ts via npx tsx.

--budget=N (default 5.0) caps THIS run only. _budget.json is a single ledger, per-project, SHARED across every target in that project, and its spentUsd persists across every run forever. --budget is compared only against this run's own fresh spend, never against that cumulative total.

--lifetime-budget=N (default: unset, no lifetime ceiling) is an OPTIONAL absolute cap across every run that ever wrote to the ledger. Use this when you want a hard stop on total project spend in addition to the per-run cap.

--holdout-repeats=N (default 2) — only takes effect for a target the engine considers nondeterministic: evalTemperature is not 0, OR the target declares nondeterministic: true (see fix #3 and L10). Ignored (no extra API calls) for deterministic targets.

--apply (default OFF): see fix #10 above. Only patches a target's declared sourceFile when the run produced a real, non-refused win and the run was not --dry-run. Every target without a declared sourceFile is unaffected regardless of this flag.

Testing

Engine tests live at engine/*.test.ts, run with vitest, provider fully mocked (no live API calls, no openai import anywhere in engine code):

npx vitest run .claude/skills/prompt-optimizer/engine --reporter=verbose