Prompt Regression Test
When To Use
A prompt edit fixed one case and quietly broke another. Run it before any change to the system prompt, a context source, a tool description or a setting goes live. It answers: did this change fix what it targeted without breaking what already worked, failure type by failure type?
When Not To Use
If the change is a new model, with a deadline and a cost difference, run Model Migration Plan. If there is no golden dataset yet, build one with Golden Dataset first; without a test set, every prompt edit is a guess.
Inputs
- The old and new prompt (or context, tool or setting), and the failure the change targets
- Results from both runs on the regression suite plus the targeted cases: pass or fail per case per trial, with the failure type of each case
If you have none of this, I start from the change and the golden dataset, write the run plan, and mark every result
[not yet run].
Approach
Anthropic Engineering's "Demystifying evals for AI agents" separates capability evals (hard tasks that start low) from regression evals (tasks the feature already handles, which should stay near 100%), and runs the regression set before every change. A change is judged per failure type, because an average can rise while a blocker gets worse. The failure it prevents: a shorter-answers edit that lifted the average and stopped the assistant from ever saying "I don't know".
Workflow
- Ask three questions: what changed and why, which failure type it targets, and which rubric checks are blockers?
- Change note: what changed (prompt, context source, tool, setting), the reason, the targeted failure, and one change only. Two changes at once make the result unreadable.
- Cases: the full regression suite plus the cases for the targeted failure type. Same cases, same trials per case, same graders before and after. In the Playground in the Claude Console you can run both versions side by side; plain chat with pasted results works too.
- Table per failure type: before pass rate, after pass rate, change, from the runs the user pasted. Synthetic cases go in a separate row, never in the headline.
- Read the flips one by one: pass to fail and fail to pass. Models are non-deterministic, so a flip can be noise; re-run flipped cases before calling them real.
- Hold rule: a rise in the average with a drop in any blocker type is a hold. A drop in a non-blocker is written down as a known regression.
- Recommendation: ship, hold, or ship with a known regression, with the reason in one line. A named person decides.
Output Format
# Prompt Regression Test
**Feature:** [name] | **Change:** [old version] to [new version] | **Decider:** [name]
## Change note
| What changed | Why | Targeted failure type |
|---|---|---|
| [prompt / context / tool / setting] | [reason] | [type] |
## Results by failure type
| Failure type | Blocker | Cases x trials | Before | After | Change |
|---|---|---|---|---|---|
| [type] | [yes / no] | [n x k] | [from run / not yet run] | [from run / not yet run] | [difference] |
| Synthetic (separate) | no | [n x k] | [from run] | [from run] | [difference] |
## Flipped cases
| Case | Before | After | Re-run result | Real or noise |
|---|---|---|---|---|
| [id] | [pass / fail] | [pass / fail] | [result] | [verdict] |
## Recommendation
[Ship / hold / ship with known regression]: [one-line reason]
## Decision
[Named person] decides to ship or hold by [date]; any known regression gets an owner and a fix date.
Done When
- One change is tested, with a change note naming the targeted failure type
- Results are per failure type, with blockers marked and synthetic cases kept apart
- Every flipped case was read and re-run before being called real
Quality Bar
- Every pass rate comes from a run the user pasted; otherwise it reads
[not yet run]
- Never one average on its own; a blocker drop overrides any average gain
- Before and after use the same cases, trials and graders, or the comparison is flagged as invalid
- Grades outputs only; no reviewer or engineer is named in a finding
- Claude compares the runs; a named person decides to ship or hold
Next
Run aipm-model-migration (Model Migration Plan) for when the change is the model itself.
About the makers
This pack is made by 90n (90n.ai): AI-native ways of working and tools for Product, Marketing and HR teams, built by ex-McKinsey founders. We map where time and money leak, redesign the work around AI and stay until your people run it themselves, and we run our own company on Claude. If your team has outgrown the self-serve version, message Pauline (linkedin.com/in/paulinebertry).
1---2name: aipm-prompt-regression-test3description: Runs a Prompt Regression Test on a prompt, context or setting change, with a change note, the cases to re-run, before and after results per failure type and a ship or hold recommendation for a named person. Use for "run aipm-prompt-regression-test", "did this prompt change break anything", "regression test a prompt edit", "before and after eval for our prompt", "the fix broke another case", "check the new system prompt", "ship or hold this prompt", part of the Claude for AI Product Managers Pack by 90n.4metadata:5 source: https://skillmd.com/skills/polar-bear-org/aipm-prompt-regression-test6---78# Prompt Regression Test910## When To Use11A prompt edit fixed one case and quietly broke another. Run it before any change to the system prompt, a context source, a tool description or a setting goes live. It answers: did this change fix what it targeted without breaking what already worked, failure type by failure type?1213## When Not To Use14If the change is a new model, with a deadline and a cost difference, run Model Migration Plan. If there is no golden dataset yet, build one with Golden Dataset first; without a test set, every prompt edit is a guess.1516## Inputs17- The old and new prompt (or context, tool or setting), and the failure the change targets18- Results from both runs on the regression suite plus the targeted cases: pass or fail per case per trial, with the failure type of each case19If you have none of this, I start from the change and the golden dataset, write the run plan, and mark every result `[not yet run]`.2021## Approach22Anthropic Engineering's "Demystifying evals for AI agents" separates capability evals (hard tasks that start low) from regression evals (tasks the feature already handles, which should stay near 100%), and runs the regression set before every change. A change is judged per failure type, because an average can rise while a blocker gets worse. The failure it prevents: a shorter-answers edit that lifted the average and stopped the assistant from ever saying "I don't know".2324## Workflow251. Ask three questions: what changed and why, which failure type it targets, and which rubric checks are blockers?262. Change note: what changed (prompt, context source, tool, setting), the reason, the targeted failure, and one change only. Two changes at once make the result unreadable.273. Cases: the full regression suite plus the cases for the targeted failure type. Same cases, same trials per case, same graders before and after. In the Playground in the Claude Console you can run both versions side by side; plain chat with pasted results works too.284. Table per failure type: before pass rate, after pass rate, change, from the runs the user pasted. Synthetic cases go in a separate row, never in the headline.295. Read the flips one by one: pass to fail and fail to pass. Models are non-deterministic, so a flip can be noise; re-run flipped cases before calling them real.306. Hold rule: a rise in the average with a drop in any blocker type is a hold. A drop in a non-blocker is written down as a known regression.317. Recommendation: ship, hold, or ship with a known regression, with the reason in one line. A named person decides.3233## Output Format34```markdown35# Prompt Regression Test36**Feature:** [name] | **Change:** [old version] to [new version] | **Decider:** [name]37## Change note38| What changed | Why | Targeted failure type |39|---|---|---|40| [prompt / context / tool / setting] | [reason] | [type] |41## Results by failure type42| Failure type | Blocker | Cases x trials | Before | After | Change |43|---|---|---|---|---|---|44| [type] | [yes / no] | [n x k] | [from run / not yet run] | [from run / not yet run] | [difference] |45| Synthetic (separate) | no | [n x k] | [from run] | [from run] | [difference] |46## Flipped cases47| Case | Before | After | Re-run result | Real or noise |48|---|---|---|---|---|49| [id] | [pass / fail] | [pass / fail] | [result] | [verdict] |50## Recommendation51[Ship / hold / ship with known regression]: [one-line reason]52## Decision53[Named person] decides to ship or hold by [date]; any known regression gets an owner and a fix date.54```5556## Done When57- One change is tested, with a change note naming the targeted failure type58- Results are per failure type, with blockers marked and synthetic cases kept apart59- Every flipped case was read and re-run before being called real6061## Quality Bar62- Every pass rate comes from a run the user pasted; otherwise it reads `[not yet run]`63- Never one average on its own; a blocker drop overrides any average gain64- Before and after use the same cases, trials and graders, or the comparison is flagged as invalid65- Grades outputs only; no reviewer or engineer is named in a finding66- Claude compares the runs; a named person decides to ship or hold6768## Next69Run aipm-model-migration (Model Migration Plan) for when the change is the model itself.7071## About the makers7273This pack is made by 90n ([90n.ai](https://90n.ai)): AI-native ways of working and tools for Product, Marketing and HR teams, built by ex-McKinsey founders. We map where time and money leak, redesign the work around AI and stay until your people run it themselves, and we run our own company on Claude. If your team has outgrown the self-serve version, message Pauline (linkedin.com/in/paulinebertry).