# AI Evals

> Use when testing an LLM-backed feature, prompt, tool loop or multi-step agent, where the same input can produce different outputs and a prompt or model change can regress behaviour with no code diff. The behaviour spec that precedes the prompt, scenario datasets including adversarial and degradation classes, deterministic assertions over OpenTelemetry traces, calibrated LLM-as-judge, CI gates with baselines, human-in-the-loop, and the production scoring loop that turns incidents into scenarios.

- Skill: `konradcinkusz/ai-evals` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add konradcinkusz/ai-evals`
- Raw SKILL.md: https://api.skillmd.com/api/skills/konradcinkusz/ai-evals/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: konradcinkusz (https://skillmd.com/u/konradcinkusz)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/konradcinkusz/ai-evals

---


# Evaluating LLM-backed features and agents

**Read [`references/AI-EVALS.md`](references/AI-EVALS.md) before applying any of this.**
That file is the standard; everything below it is a summary to help you decide
whether this skill applies and to check your work afterwards.

Reference-architecture principles: P13.

## What this standard covers

- The model in one paragraph
- The spec comes first
- Scenarios are data
- Layer 1: deterministic assertions on traces
- Layer 2: LLM-as-judge
- Gates
- Production scoring closes the loop
- Human-in-the-loop
- Prove the suite can fail

## Failure modes

| Symptom | Cause |
|---|---|
| Suite green, agent visibly worse in production | Scenarios cover the demo paths; no incident-to-scenario loop; judge grading fluency, not grounding |
| Constraint scenario "passes" on a broken agent | Guard-then-bail before the assertion; the agent did nothing and the eval skipped |
| Eval results differ run to run with no change | Nondeterminism unpinned: temperature, model minor version, or fixture state not reset between scenarios |
| Judge scores jump after a quiet week | Judge model or prompt changed without a suite version bump and re-baseline |
| Judge and users disagree about quality | Judge never calibrated against human labels; rubric anchors too vague to grade against |
| Prompt edit ships with no eval run | Change detection watches code paths only; prompts and agent definitions not mapped as eval-triggering paths |
| Adversarial scenarios pass, production injection succeeds | Injection tested only via user input; tool-result injection path uncovered |
| Eval suite abandoned within a quarter | No budget split: full judge matrix on every PR priced the suite out of the loop |
| Confirmation bypass found by a user, not a page | Constraint checks run offline only; no post-hoc verification over production traces |
| "It works" defended from one good transcript | No baseline recorded; anecdote standing in for a pass-rate diff |
| A new write tool ships with no constraint on it, suite still green | Write-classification inferred from a name prefix instead of a normative table, so the unnamed tool was silently classified a read (§4) |
| A fixture edit lands unreviewed, and a scenario now measures something else | Per-scenario worlds copied wholesale instead of a base plus a delta, so the change is spread across near-identical files rather than visible in one (§3) |
| A suite reports 100% for a year and has never been shown able to fail | No mutation pass: real assertions prove they can pass, never that they can catch (§9) |
| A mutation variant "caught" by a harness crash | The pass counted an error as a catch, so the rate measures the harness's ability to throw (§9) |

## Checklist

Per agent or LLM-backed feature:

- [ ] Behaviour spec in-repo, versioned with the agent definition: behaviours, hard constraints, success criteria, out-of-scope — with negatives stated
- [ ] Scenario dataset as data, covering happy / ambiguity / denied / adversarial (both injection paths) / degradation classes
- [ ] Fixtures are a named base world plus a per-scenario delta, rebuilt from scratch each run — never mutated in place, never one full world copied per scenario
- [ ] Agent loop instrumented per OTel GenAI conventions; confirmations are trace events
- [ ] Layer 1 asserts over traces: right calls, right arguments, ordering, absence, termination — no guard-then-bail, no swallowed failures, unimplemented = `Skip`
- [ ] Write-classification pinned as a normative per-tool table in the spec and derived from it by the harness — never from a naming convention
- [ ] Layer 2: rubric-anchored per-criterion judge that sees the trace; judge model + prompt pinned and versioned; calibration against human labels recorded before scores gate
- [ ] Gates per §6: constraints hard-block at 100%; behaviour vs baseline; judge thresholds; prompts and definitions included in change detection
- [ ] Nightly matrix with baseline diffs; PR output is a diff, not a dashboard
- [ ] Production sessions scored on the shared trace schema; worst sessions read on a cadence; low scorers converted to scenarios; constraint checks run post-hoc with paging
- [ ] Human review sampled and scheduled; findings become scenarios and rubric fixes
- [ ] Every production incident has a scenario before it has a fix
- [ ] Deliberately broken agent variants exist and the constraint layer catches every one; a surviving variant is filed as a missing scenario, and a harness crash never counts as a catch

---

Generated from [`docs/guides/AI-EVALS.md`](https://github.com/konradcinkusz/architecture-standards/blob/main/docs/guides/AI-EVALS.md) by `scripts/build-marketplace.mjs`. Do not edit this file: change the source document, or its entry in `catalog/marketplace.catalog.json`, and re-run the generator.