Prompt Regression Gate AI Skill Guide (Claude)
Overview & Engine Architecture
A prompt is production code. Unit tests do not catch “fluent but worse.” A regression gate answers only: is this change worse than the pinned baseline on the same frozen cases? Absolute “looks good” review and candidate self-confidence are not gates.
Claude operates as a Principal LLM Release Engineer, specializing in paired baseline vs candidate runs, relative delta thresholds, slice/P0 hard fails, stochastic control, and hash-bound provenance (prompt, dataset, system, judge).
┌─────────────────────────────────────────────────────────────┐
│ Prompt Regression Gate │
│ │
│ Inputs (frozen before compare) │
│ ├── dataset_version + dataset_hash (@ai-evaluation-dataset)│
│ ├── baseline: prompt_hash + model + tools + retriever │
│ ├── candidate: same knobs except the intentional delta │
│ └── thresholds agreed BEFORE seeing scores │
│ │
│ Runner │
│ ├── Paired per-case execution (identical context) │
│ ├── Deterministic scorers + optional pinned LLM judge │
│ └── Provider errors tracked separately from quality fails │
│ │
│ Gate │
│ ├── Aggregate delta vs baseline within tolerance │
│ ├── Slice / P0: no critical regressions │
│ └── Verdict: go | no-go | untested + rollback plan │
└─────────────────────────────────────────────────────────────┘
When to use / when not to
Use when
- A PR touches prompts, templates, tool/JSON schemas, packers, model IDs, or retrieval config that changes model context.
- Promoting a model upgrade or temperature/tool change.
- You need a merge-blocking CI check, not a dashboard vibe.
Do not use when
- There is no frozen eval set yet — build
@ai-evaluation-datasetfirst. - Someone wants to “tune until the gate passes” on the held-out set (that destroys the audit).
- Claiming a winner without executed paired runs (deliver config only; mark untested).
Operational Capabilities & Agent Directives
- Agree thresholds before scores — write them in the PR/template; do not move bars after seeing results.
- One intentional delta — change prompt or model or tools or retriever; log everything else as identical.
- Reuse held-out / golden cases — never add failing test cases into the gate set mid-experiment to force a pass; promote failures via a new dataset version after adjudication.
- Pair cases — same
case_id, same retrieved context / tool stubs unless retrieval is the variable under test. - Pin judges — exact judge model + judge prompt hash; temp 0 (or documented sampling with repeats).
- Never use the candidate’s self-reported confidence as the sole quality metric.
- Blind humans to baseline/candidate identity when rubric review is required.
- Separate provider/timeouts/rate-limits from “wrong answer” in metrics.
What must be captured (run manifest)
{
"gate_id": "support-agent-prompt",
"dataset_version": "v3",
"dataset_hash": "sha256:…",
"baseline": {
"prompt_hash": "sha256:…",
"model": "gpt-4.1-2025-04-14",
"tools_hash": "sha256:…",
"retriever_revision": "idx-…",
"gen": {"temperature": 0, "max_tokens": 1024}
},
"candidate": {
"prompt_hash": "sha256:…",
"model": "gpt-4.1-2025-04-14",
"tools_hash": "sha256:…",
"retriever_revision": "idx-…",
"gen": {"temperature": 0, "max_tokens": 1024}
},
"judge": {"model": "…", "prompt_hash": "sha256:…", "temperature": 0},
"thresholds": {
"max_aggregate_drop": 0.02,
"max_slice_drop": 0.03,
"p0_allowed_regressions": 0,
"min_decisive_cases_for_claim": 50
},
"budget": {"max_usd": 25, "repeats": 1}
}
Hash prompt files as canonical bytes (normalized newlines). Include system + developer + tool descriptors in prompt_hash / tools_hash.
Scoring strategy
| Output type | Prefer | Notes |
|---|---|---|
| JSON / tools / enums | Deterministic (@llm-json-contract-check, exact/regex) |
Cheap, stable CI |
| Citations / grounding | Deterministic resolve-to-context | Don’t LLM-judge retrieval arithmetic |
| Open-ended quality | Rubric or pinned LLM judge | Average ≥2 judge passes if noisy |
| Safety / refusal | Dedicated slice + hard gate | Average gains must not hide P0 |
Relative gate (recommended): fail if score_candidate < score_baseline - tolerance (often 0.02–0.05 absolute pass-rate or mean rubric). Absolute floors (“must be ≥90%”) are optional secondary bars and go stale.
Per-case first-class outcomes
improved/regressed/tied/error_baseline/error_candidate- Slice tags from the dataset:
p0,refusal,ambiguous,adversarial,rag,latency_sensitive
A small average gain that hides a P0 regression is a no-go.
Statistical honesty (don’t overclaim)
On paired decisive cases (exclude ties), a coin-flip null is ~50/50 wins. Rough two-sided sign-test intuition (llmbestpractices):
- ~100 decisive cases → need ~61 wins for p<0.05
- ~50 decisive cases → need ~33 wins
Report sample size and uncertainty. “+2pp on 40 cases” is usually noise—not a promotion story.
If outputs are stochastic and budget allows: repeat each case N times (e.g. 3) and majority-vote or average before compare.
Procedure
1. Preflight (block if missing)
- Frozen dataset + hash; thresholds; baseline artifact from main (not another open PR)
- Secrets for providers; cost budget
- Path filter intent: only run when prompts/evals/LLM call paths change
2. Execute paired runs
- Load cases; for each
case_id, run baseline then candidate (or parallel with isolated caches). - Score with the same scorers/judge revision.
- Record missing runs and HTTP/provider errors separately.
- Persist per-case JSONL + summary.
3. Apply gate rules (order matters)
- Infra: if candidate error rate exceeds agreed bound → no-go (or retry once).
- P0 / critical slice: any regression beyond tolerance → no-go.
- Aggregate: relative drop beyond
max_aggregate_drop→ no-go. - Optional: latency/cost ceilings (
@llm-cost-latency-benchmark). - Else go, with failing/improving example links.
4. Deliverable
Always produce:
- Go / no-go / untested (if runs did not execute)
- Aggregate + slice table (baseline, candidate, delta)
- Top regressed
case_ids with short diffs - Provenance hashes (dataset, prompts, judge, system)
- Rollback: revert prompt file / model pin / feature flag instructions
After merge on main: refresh baseline artifact from the new main run (explicit promotion).
Production example: relative gate comparator
"""Compare baseline vs candidate per-case scores; exit 1 on regression."""
from __future__ import annotations
import json
import sys
from dataclasses import dataclass
from pathlib import Path
from statistics import fmean
from typing import Any
@dataclass(frozen=True)
class Thresholds:
max_aggregate_drop: float = 0.02
max_slice_drop: float = 0.03
p0_allowed_regressions: int = 0
error_rate_max: float = 0.05
def load_scores(path: Path) -> dict[str, dict[str, Any]]:
"""JSONL rows: case_id, score (0..1), slice (list[str]), error (bool)."""
out: dict[str, dict[str, Any]] = {}
for line in path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
row = json.loads(line)
out[row["case_id"]] = row
return out
def mean_score(rows: dict[str, dict[str, Any]], ids: list[str] | None = None) -> float:
keys = ids or list(rows)
vals = [rows[i]["score"] for i in keys if not rows[i].get("error")]
return fmean(vals) if vals else float("nan")
def gate(baseline_path: Path, candidate_path: Path, th: Thresholds) -> int:
b, c = load_scores(baseline_path), load_scores(candidate_path)
ids = sorted(set(b) & set(c))
if not ids:
print("NO-GO: no paired case ids")
return 2
b_err = sum(1 for i in ids if b[i].get("error")) / len(ids)
c_err = sum(1 for i in ids if c[i].get("error")) / len(ids)
if c_err > th.error_rate_max:
print(f"NO-GO: candidate error rate {c_err:.2%} > {th.error_rate_max:.2%}")
return 1
# P0 regressions: baseline pass-ish, candidate worse beyond tiny eps
p0_regs = []
for i in ids:
slices = set(b[i].get("slice") or []) | set(c[i].get("slice") or [])
if "p0" not in slices:
continue
if b[i].get("error") or c[i].get("error"):
continue
if c[i]["score"] + 1e-9 < b[i]["score"]:
p0_regs.append(i)
if len(p0_regs) > th.p0_allowed_regressions:
print(f"NO-GO: P0 regressions: {p0_regs}")
return 1
b_mean, c_mean = mean_score(b, ids), mean_score(c, ids)
drop = b_mean - c_mean
print(f"aggregate baseline={b_mean:.4f} candidate={c_mean:.4f} drop={drop:.4f}")
if drop > th.max_aggregate_drop:
print("NO-GO: aggregate regression")
return 1
# slice checks
slice_names = sorted({s for i in ids for s in (b[i].get("slice") or [])})
for name in slice_names:
sids = [i for i in ids if name in (b[i].get("slice") or [])]
sd = mean_score(b, sids) - mean_score(c, sids)
print(f"slice {name}: drop={sd:.4f} n={len(sids)}")
if sd > th.max_slice_drop:
print(f"NO-GO: slice regression ({name})")
return 1
# paired win counts (exclude ties & errors)
wins = loses = 0
for i in ids:
if b[i].get("error") or c[i].get("error"):
continue
if c[i]["score"] > b[i]["score"] + 1e-9:
wins += 1
elif c[i]["score"] + 1e-9 < b[i]["score"]:
loses += 1
print(f"paired decisive wins={wins} loses={loses} (ties excluded)")
print("GO")
return 0
if __name__ == "__main__":
sys.exit(gate(Path(sys.argv[1]), Path(sys.argv[2]), Thresholds()))
CI sketch (path-filtered)
# Run only when prompts / evals / LLM callers change
on:
pull_request:
paths: ["prompts/**", "evals/**", "src/**/llm/**", ".github/workflows/prompt-gate.yml"]
# 1) Run candidate suite → candidate.jsonl
# 2) Fetch baseline.jsonl from main branch artifact or committed baseline
# 3) python scripts/prompt_gate.py baseline.jsonl candidate.jsonl
# 4) On main merge: refresh baseline artifact
Compatible runners: Promptfoo (eval --fail-on-error + custom delta), Evalgate-style baseline compare, DeepEval for RAG slices—same relative question.
Technical troubleshooting matrix
| Signature | Root cause | Fix |
|---|---|---|
| Flaky red/green | temp>0, unpinned judge, tiny N | temp 0; pin judge; more cases/repeats; tolerance margin |
| “Improved” after editing gold | Dataset moved mid-gate | Immutable dataset; bridge via @ai-evaluation-dataset |
| Baseline from another PR | Wrong artifact selection | Only ci/main baselines |
| Avg up, safety down | No slice gates | Hard-fail p0 / refusal slices |
| README PR burns $ | No path filter | Limit workflow paths |
| Untested marked GO | Skipped runner | Forbid; status must be untested/no-go |
| Model upgrade blamed on prompt | Multiple deltas | Single-variable compare; re-baseline deliberately |
Best practices
- Treat prompts as versioned files with hashes, not ChatGPT scratchpads.
- Grow the set from production failures; keep adversarial/refusal as visible slices.
- Prefer deterministic scorers in CI; LLM judges for open-ended with pins + averaging.
- Update eval cases in the same commit when output contracts/schemas change.
- Re-baseline consciously when upgrading the judge or dataset version—never silently.
- Post a PR comment table (baseline / head / delta / slice)—humans review regressions, CI blocks merges.
Limitations
- Gates measure the frozen set, not the whole world—pair with production holdout monitoring.
- This skill does not replace red-team security testing (
@agent-injection-boundary-test). - Stop and ask if thresholds, baseline source, or dataset hash are missing.
Related skills
@ai-evaluation-dataset— frozen cases + hashes@rag-retrieval-audit— isolate retrieval before prompt gates@llm-json-contract-check— deterministic structured scorers@llm-cost-latency-benchmark— cost/latency ceilings in the same gate@ai-citation-verification— grounding assertions@prompt-regression-gate— this skill@ai-human-handoff-contract— P0 human paths
Agent Operational Directive
MANDATORY: Gate with paired baseline vs candidate on a frozen hashed dataset. Agree thresholds before scores. Fail on P0/slice regressions even if the average rises. Pin judge and generation settings. Record provider errors separately. Without executed runs, mark untested—never invent a go. Deliver rollback instructions with every no-go.
Source anchors (research)
- Prompt evals and regression testing (paired runs, sign-test caution, don’t tune on gold)
- LLM evals in CI/CD — relative gates
- Promptfoo CI/CD
- Evalgate / prompt-regress patterns: baseline delta, PR comments, main-only baselines
- Agent eval templates: Promptfoo + DeepEval + regression delta threshold