# Nuke Eval

> Use when a prompt, system prompt, agent instruction, or LLM-app behavior changes and must be proven, not vibed — "nuke eval", "eval this prompt", "did my prompt change help", "add evals" — or when a repo ships prompts with no eval harness at all. No prompt change is "done" without an eval run; evals themselves are proven by perturbation.

- Skill: `b4r7x/nuke-eval` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add b4r7x/nuke-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/b4r7x/nuke-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: b4r7x (https://skillmd.com/u/b4r7x)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/b4r7x/nuke-eval

---


# Nuke Eval

nuke-test for model behavior (map: references/family-map.md — nuke-prompt's breaker findings arrive here as adversarial cases). A prompt edit is a code change whose gates nobody runs: someone reorders a sentence, output "feels better", and the regression ships. This skill maps what the prompt promises, builds an eval set, runs it before/after, and proves the evals themselves by perturbation — the mutation check's sibling: an eval no prompt-degradation can fail is vacuous and gets deleted.

## Arguments

`[mode]` — `light` (default) | `full` | `plan` (preflight, print the plan block, STOP).
`[target]` — a prompt/instruction file, an agent directory, or `changed` (prompt-bearing files changed vs HEAD, default).
`--ask` — pause at the preflight plan for confirmation; default is no gate — the plan prints and the run starts (references/preflight.md).

## Modes

Tier vocabulary and platform mechanics: references/model-tiers.md.

| | **light** (default) | **full** |
|---|---|---|
| Behavior mapper | session | session |
| Case writers | worker | worker |
| Judge design | mechanical asserts first; LLM-judge only where asserts cannot reach | same, LLM-judge rubrics reviewed at top |
| Perturbation check | 3–5 perturbations, riskiest behaviors | ≥1 per newly covered behavior |
| Runs per case | 1 (flag nondeterminism when seen) | 3 — a case must be stable or its flakiness is a finding |
| Cost stance | lean: riskiest behaviors first | costs do not matter; never shrink the map |

## Mandates

1. **An eval that cannot fail is a defect.** The perturbation check is the proof: degrade the prompt deliberately (drop a constraint line, invert an instruction, remove a few-shot) — the evals covering that behavior must fail. Survivors are coverage gaps; vacuous evals are deleted, never kept.
2. **Behaviors come from the prompt's contract** — what it promises: output format, refusals, tone, task success, constraint adherence — read from the prompt text and its call sites, never invented.
3. **Mechanical asserts before LLM judges.** Format checks, schema validation, string/regex presence, length bounds first; an LLM-judge rubric only for what asserts cannot reach, and the rubric itself states pass/fail criteria a second judge could apply identically.
4. **Never leave a perturbation applied.** One at a time; revert verified before the next; the final tree contains only eval files and the baseline report.
5. **Model calls cost money — say so.** The preflight estimate includes eval-run API calls (cases × runs × cycles), not just agent count.
6. **No `git add` / `git commit` / `git stash`.** Working tree left for review.
7. **Never claim improvement without both numbers.** Before/after pass rates from the same eval set, same model, same settings — or the claim is not made.

## Artifacts

`run_dir = .nuke/<YYYY-MM-DD>-<HHmmss>-eval-<slug>/` — fresh per run; on collision append `-2`, `-3`, ….

| File | Role |
|---|---|
| `plan.md` | Preflight output: target, prompt inventory, harness, wave, tiers, API-call estimate |
| `behaviors.md` | Numbered, risk-ranked promise map with coverage marks |
| `evals/` | The eval set: cases + runner (kept in-repo only if the user confirms a destination in preflight) |
| `results.md` | Baseline scores, perturbation verdicts, before/after when a change is under test |

## Pipeline

```
Phase 0 preflight (plan → apply) → Phase 1 promise map (session) → Phase 2 case writers (worker) → Phase 3 perturbation check → Phase 4 baseline/before-after + report (STOP)
                                                     └── surviving perturbation → Phase 2 for that behavior — cap 2 cycles ──┘
```

## Phase 0 — Preflight

Read references/preflight.md and follow its plan-then-apply gate. Eval specifics:

1. Resolve the target into a **prompt inventory**: every file whose content reaches a model (system prompts, instruction blocks, templates, few-shots, tool descriptions). `changed` resolving to zero prompt-bearing files → STOP and ask.
2. **Harness resolution:** find the existing eval setup (eval scripts, test files calling models, promptfoo/braintrust-class configs). None → the plan proposes a minimal runner (one script, cases as data) and asks where it should live.
3. **Existing-eval inventory** — the dedup baseline.
4. Estimate agents AND model calls (Mandate 5). Print the plan block and apply it — gate only with `--ask`. `plan` → STOP.

## Phase 1 — Promise map (session)

One mapper reads the prompt inventory and its call sites, and writes `behaviors.md`: numbered `P-##` promises, one observable behavior each, risk-ranked — task success on the critical path first, then refusal/safety behavior, then format contracts, then tone/style. Each marked `covered` (existing eval cited) or `uncovered`. Sources are the prompt text and call sites only (Mandate 2).

## Phase 2 — Case writers (worker)

Writers in parallel, ≤5 promises each, disjoint case files. Every writer prompt embeds: its `P-##` entries verbatim · the harness contract · the assert-first rule (Mandate 3) · this rule-set:

1. Golden cases: representative inputs with mechanically checkable expected properties — never exact-string-match on free prose.
2. Adversarial cases per promise: the input most likely to break it (edge formats, injection attempts against tool-use prompts, instructions that conflict with the system prompt).
3. One promise per case; the case name states the promise.
4. A case that passes against the current prompt AND an obviously degraded one is not yet a case.

## Phase 3 — Perturbation check

The breaker (session light / top full) proves the set: propose perturbations (`X-##`) — each targets one promise: drop its constraint line, invert it, delete the relevant few-shot, truncate the prompt. Apply one at a time to a COPY of the prompt wired into the harness; run the covering cases; **killed** (case fails) proves coverage, **survived** sends the promise back to Phase 2 with the perturbation quoted. Revert verified per Mandate 4. Cap 2 cycles; survivors after cycle 2 are reported gaps.

## Phase 4 — Baseline / before-after + report (STOP)

1. Run the full set against the current prompt → baseline scores into results.md. When the run's purpose was judging a specific prompt change: run against both versions, report per-promise pass rates side by side, verbatim.
2. Report: promise map coverage, perturbation scorecard (killed/survived), vacuous evals deleted, baseline (or before/after) numbers, honest gaps, artifact paths, calibration line appended to `.nuke/calibration.log` (format in references/preflight.md).
3. STOP. Regressions found in the prompt itself → recommend nuke-prompt for a disciplined rewrite; never rewrite here.

## Orchestration notes

- Writers parallel on disjoint files; the breaker alone, perturbations strictly serial. Tiers per references/model-tiers.md; no per-agent overrides → every role inherits the session model.
- Platforms without a way to call the target model from the harness (no API key, no CLI) → build the case set anyway, mark `results: NOT RUN — no model access`, and say exactly what command runs it later. Never fake scores.