Nuke Eval

Nuke Eval

nuke-test for model behavior (map: references/family-map.md — nuke-prompt's breaker findings arrive here as adversarial cases). A prompt edit is a code change whose gates nobody runs: someone reorders a sentence, output "feels better", and the regression ships. This skill maps what the prompt promises, builds an eval set, runs it before/after, and proves the evals themselves by perturbation — the mutation check's sibling: an eval no prompt-degradation can fail is vacuous and gets deleted.

Arguments

[mode] — light (default) | full | plan (preflight, print the plan block, STOP). [target] — a prompt/instruction file, an agent directory, or changed (prompt-bearing files changed vs HEAD, default). --ask — pause at the preflight plan for confirmation; default is no gate — the plan prints and the run starts (references/preflight.md).

Modes

Tier vocabulary and platform mechanics: references/model-tiers.md.

Mandates

1. An eval that cannot fail is a defect. The perturbation check is the proof: degrade the prompt deliberately (drop a constraint line, invert an instruction, remove a few-shot) — the evals covering that behavior must fail. Survivors are coverage gaps; vacuous evals are deleted, never kept.
2. Behaviors come from the prompt's contract — what it promises: output format, refusals, tone, task success, constraint adherence — read from the prompt text and its call sites, never invented.
3. Mechanical asserts before LLM judges. Format checks, schema validation, string/regex presence, length bounds first; an LLM-judge rubric only for what asserts cannot reach, and the rubric itself states pass/fail criteria a second judge could apply identically.
4. Never leave a perturbation applied. One at a time; revert verified before the next; the final tree contains only eval files and the baseline report.
5. Model calls cost money — say so. The preflight estimate includes eval-run API calls (cases × runs × cycles), not just agent count.
6. No git add / git commit / git stash. Working tree left for review.
7. Never claim improvement without both numbers. Before/after pass rates from the same eval set, same model, same settings — or the claim is not made.

Artifacts

run_dir = .nuke/<YYYY-MM-DD>-<HHmmss>-eval-<slug>/ — fresh per run; on collision append -2, -3, ….

Pipeline

Phase 0 preflight (plan → apply) → Phase 1 promise map (session) → Phase 2 case writers (worker) → Phase 3 perturbation check → Phase 4 baseline/before-after + report (STOP)
                                                     └── surviving perturbation → Phase 2 for that behavior — cap 2 cycles ──┘

Phase 0 — Preflight

Read references/preflight.md and follow its plan-then-apply gate. Eval specifics:

1. Resolve the target into a prompt inventory: every file whose content reaches a model (system prompts, instruction blocks, templates, few-shots, tool descriptions). changed resolving to zero prompt-bearing files → STOP and ask.
2. Harness resolution: find the existing eval setup (eval scripts, test files calling models, promptfoo/braintrust-class configs). None → the plan proposes a minimal runner (one script, cases as data) and asks where it should live.
3. Existing-eval inventory — the dedup baseline.
4. Estimate agents AND model calls (Mandate 5). Print the plan block and apply it — gate only with --ask. plan → STOP.

Phase 1 — Promise map (session)

One mapper reads the prompt inventory and its call sites, and writes behaviors.md: numbered P-## promises, one observable behavior each, risk-ranked — task success on the critical path first, then refusal/safety behavior, then format contracts, then tone/style. Each marked covered (existing eval cited) or uncovered. Sources are the prompt text and call sites only (Mandate 2).

Phase 2 — Case writers (worker)

Writers in parallel, ≤5 promises each, disjoint case files. Every writer prompt embeds: its P-## entries verbatim · the harness contract · the assert-first rule (Mandate 3) · this rule-set:

1. Golden cases: representative inputs with mechanically checkable expected properties — never exact-string-match on free prose.
2. Adversarial cases per promise: the input most likely to break it (edge formats, injection attempts against tool-use prompts, instructions that conflict with the system prompt).
3. One promise per case; the case name states the promise.
4. A case that passes against the current prompt AND an obviously degraded one is not yet a case.

Phase 3 — Perturbation check

The breaker (session light / top full) proves the set: propose perturbations (X-##) — each targets one promise: drop its constraint line, invert it, delete the relevant few-shot, truncate the prompt. Apply one at a time to a COPY of the prompt wired into the harness; run the covering cases; killed (case fails) proves coverage, survived sends the promise back to Phase 2 with the perturbation quoted. Revert verified per Mandate 4. Cap 2 cycles; survivors after cycle 2 are reported gaps.

Phase 4 — Baseline / before-after + report (STOP)

1. Run the full set against the current prompt → baseline scores into results.md. When the run's purpose was judging a specific prompt change: run against both versions, report per-promise pass rates side by side, verbatim.
2. Report: promise map coverage, perturbation scorecard (killed/survived), vacuous evals deleted, baseline (or before/after) numbers, honest gaps, artifact paths, calibration line appended to .nuke/calibration.log (format in references/preflight.md).
3. STOP. Regressions found in the prompt itself → recommend nuke-prompt for a disciplined rewrite; never rewrite here.

Orchestration notes

• Writers parallel on disjoint files; the breaker alone, perturbations strictly serial. Tiers per references/model-tiers.md; no per-agent overrides → every role inherits the session model.
• Platforms without a way to call the target model from the harness (no API key, no CLI) → build the case set anyway, mark results: NOT RUN — no model access, and say exactly what command runs it later. Never fake scores.