Use when testing an LLM-backed feature, prompt, tool loop or multi-step agent, where the same input can produce different outputs and a prompt or model change can regress behaviour with no code diff. The behaviour spec that precedes the prompt, scenario datasets including adversarial and degradation classes, deterministic assertions over OpenTelemetry traces, calibrated LLM-as-judge, CI gates with baselines, human-in-the-loop, and the production scoring loop that turns incidents into scenarios.
Read references/AI-EVALS.md before applying any of this.
That file is the standard; everything below it is a summary to help you decide
whether this skill applies and to check your work afterwards.
Reference-architecture principles: P13.
What this standard covers
The model in one paragraph
The spec comes first
Scenarios are data
Layer 1: deterministic assertions on traces
Layer 2: LLM-as-judge
Gates
Production scoring closes the loop
Human-in-the-loop
Prove the suite can fail
Failure modes
Symptom
Cause
Suite green, agent visibly worse in production
Scenarios cover the demo paths; no incident-to-scenario loop; judge grading fluency, not grounding
Constraint scenario "passes" on a broken agent
Guard-then-bail before the assertion; the agent did nothing and the eval skipped
Eval results differ run to run with no change
Nondeterminism unpinned: temperature, model minor version, or fixture state not reset between scenarios
Judge scores jump after a quiet week
Judge model or prompt changed without a suite version bump and re-baseline
Judge and users disagree about quality
Judge never calibrated against human labels; rubric anchors too vague to grade against
Prompt edit ships with no eval run
Change detection watches code paths only; prompts and agent definitions not mapped as eval-triggering paths
Adversarial scenarios pass, production injection succeeds
Injection tested only via user input; tool-result injection path uncovered
Eval suite abandoned within a quarter
No budget split: full judge matrix on every PR priced the suite out of the loop
Confirmation bypass found by a user, not a page
Constraint checks run offline only; no post-hoc verification over production traces
"It works" defended from one good transcript
No baseline recorded; anecdote standing in for a pass-rate diff
A new write tool ships with no constraint on it, suite still green
Write-classification inferred from a name prefix instead of a normative table, so the unnamed tool was silently classified a read (§4)
A fixture edit lands unreviewed, and a scenario now measures something else
Per-scenario worlds copied wholesale instead of a base plus a delta, so the change is spread across near-identical files rather than visible in one (§3)
A suite reports 100% for a year and has never been shown able to fail
No mutation pass: real assertions prove they can pass, never that they can catch (§9)
A mutation variant "caught" by a harness crash
The pass counted an error as a catch, so the rate measures the harness's ability to throw (§9)
Checklist
Per agent or LLM-backed feature:
Behaviour spec in-repo, versioned with the agent definition: behaviours, hard constraints, success criteria, out-of-scope — with negatives stated
Fixtures are a named base world plus a per-scenario delta, rebuilt from scratch each run — never mutated in place, never one full world copied per scenario
Agent loop instrumented per OTel GenAI conventions; confirmations are trace events
Layer 1 asserts over traces: right calls, right arguments, ordering, absence, termination — no guard-then-bail, no swallowed failures, unimplemented = Skip
Write-classification pinned as a normative per-tool table in the spec and derived from it by the harness — never from a naming convention
Layer 2: rubric-anchored per-criterion judge that sees the trace; judge model + prompt pinned and versioned; calibration against human labels recorded before scores gate
Gates per §6: constraints hard-block at 100%; behaviour vs baseline; judge thresholds; prompts and definitions included in change detection
Nightly matrix with baseline diffs; PR output is a diff, not a dashboard
Production sessions scored on the shared trace schema; worst sessions read on a cadence; low scorers converted to scenarios; constraint checks run post-hoc with paging
Human review sampled and scheduled; findings become scenarios and rubric fixes
Every production incident has a scenario before it has a fix
Deliberately broken agent variants exist and the constraint layer catches every one; a surviving variant is filed as a missing scenario, and a harness crash never counts as a catch
Generated from docs/guides/AI-EVALS.md by scripts/build-marketplace.mjs. Do not edit this file: change the source document, or its entry in catalog/marketplace.catalog.json, and re-run the generator.
1---2name: ai-evals3description: Use when testing an LLM-backed feature, prompt, tool loop or multi-step agent, where the same input can produce different outputs and a prompt or model change can regress behaviour with no code diff. The behaviour spec that precedes the prompt, scenario datasets including adversarial and degradation classes, deterministic assertions over OpenTelemetry traces, calibrated LLM-as-judge, CI gates with baselines, human-in-the-loop, and the production scoring loop that turns incidents into scenarios.4---56# Evaluating LLM-backed features and agents78**Read [`references/AI-EVALS.md`](references/AI-EVALS.md) before applying any of this.**9That file is the standard; everything below it is a summary to help you decide10whether this skill applies and to check your work afterwards.1112Reference-architecture principles: P13.1314## What this standard covers1516- The model in one paragraph17- The spec comes first18- Scenarios are data19- Layer 1: deterministic assertions on traces20- Layer 2: LLM-as-judge21- Gates22- Production scoring closes the loop23- Human-in-the-loop24- Prove the suite can fail2526## Failure modes2728| Symptom | Cause |29|---|---|30| Suite green, agent visibly worse in production | Scenarios cover the demo paths; no incident-to-scenario loop; judge grading fluency, not grounding |31| Constraint scenario "passes" on a broken agent | Guard-then-bail before the assertion; the agent did nothing and the eval skipped |32| Eval results differ run to run with no change | Nondeterminism unpinned: temperature, model minor version, or fixture state not reset between scenarios |33| Judge scores jump after a quiet week | Judge model or prompt changed without a suite version bump and re-baseline |34| Judge and users disagree about quality | Judge never calibrated against human labels; rubric anchors too vague to grade against |35| Prompt edit ships with no eval run | Change detection watches code paths only; prompts and agent definitions not mapped as eval-triggering paths |36| Adversarial scenarios pass, production injection succeeds | Injection tested only via user input; tool-result injection path uncovered |37| Eval suite abandoned within a quarter | No budget split: full judge matrix on every PR priced the suite out of the loop |38| Confirmation bypass found by a user, not a page | Constraint checks run offline only; no post-hoc verification over production traces |39| "It works" defended from one good transcript | No baseline recorded; anecdote standing in for a pass-rate diff |40| A new write tool ships with no constraint on it, suite still green | Write-classification inferred from a name prefix instead of a normative table, so the unnamed tool was silently classified a read (§4) |41| A fixture edit lands unreviewed, and a scenario now measures something else | Per-scenario worlds copied wholesale instead of a base plus a delta, so the change is spread across near-identical files rather than visible in one (§3) |42| A suite reports 100% for a year and has never been shown able to fail | No mutation pass: real assertions prove they can pass, never that they can catch (§9) |43| A mutation variant "caught" by a harness crash | The pass counted an error as a catch, so the rate measures the harness's ability to throw (§9) |4445## Checklist4647Per agent or LLM-backed feature:4849- [ ] Behaviour spec in-repo, versioned with the agent definition: behaviours, hard constraints, success criteria, out-of-scope — with negatives stated50- [ ] Scenario dataset as data, covering happy / ambiguity / denied / adversarial (both injection paths) / degradation classes51- [ ] Fixtures are a named base world plus a per-scenario delta, rebuilt from scratch each run — never mutated in place, never one full world copied per scenario52- [ ] Agent loop instrumented per OTel GenAI conventions; confirmations are trace events53- [ ] Layer 1 asserts over traces: right calls, right arguments, ordering, absence, termination — no guard-then-bail, no swallowed failures, unimplemented = `Skip`54- [ ] Write-classification pinned as a normative per-tool table in the spec and derived from it by the harness — never from a naming convention55- [ ] Layer 2: rubric-anchored per-criterion judge that sees the trace; judge model + prompt pinned and versioned; calibration against human labels recorded before scores gate56- [ ] Gates per §6: constraints hard-block at 100%; behaviour vs baseline; judge thresholds; prompts and definitions included in change detection57- [ ] Nightly matrix with baseline diffs; PR output is a diff, not a dashboard58- [ ] Production sessions scored on the shared trace schema; worst sessions read on a cadence; low scorers converted to scenarios; constraint checks run post-hoc with paging59- [ ] Human review sampled and scheduled; findings become scenarios and rubric fixes60- [ ] Every production incident has a scenario before it has a fix61- [ ] Deliberately broken agent variants exist and the constraint layer catches every one; a surviving variant is filed as a missing scenario, and a harness crash never counts as a catch6263---6465Generated from [`docs/guides/AI-EVALS.md`](https://github.com/konradcinkusz/architecture-standards/blob/main/docs/guides/AI-EVALS.md) by `scripts/build-marketplace.mjs`. Do not edit this file: change the source document, or its entry in `catalog/marketplace.catalog.json`, and re-run the generator.
Run npx skillmds@latest add konradcinkusz/ai-evals in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.
Use when testing an LLM-backed feature, prompt, tool loop or multi-step agent, where the same input can produce different outputs and a prompt or model change can regress behaviour with no code diff. The behaviour spec that precedes the prompt, scenario datasets including adversarial and degradation classes, deterministic assertions over OpenTelemetry traces, calibrated LLM-as-judge, CI gates with baselines, human-in-the-loop, and the production scoring loop that turns incidents into scenarios. It is listed under AI & ML on SkillMD.
This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.
This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.
Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.
konradcinkusz (@konradcinkusz) published this skill. Their other Agent Skills are listed on their SkillMD profile.
konradcinkusz