# AI Eval Harness

> Designs and runs evaluation harnesses for AI features and for agent skills themselves. Use when shipping LLM features or measuring skill quality. Emits EVAL_PLAN / EVAL_REPORT. Never claims production quality from a handful of vibes-only prompts.

- Skill: `willianbs/ai-eval-harness` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add willianbs/ai-eval-harness`
- Raw SKILL.md: https://api.skillmd.com/api/skills/willianbs/ai-eval-harness/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: willianbs (https://skillmd.com/u/willianbs)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/willianbs/ai-eval-harness

---


# Purpose

Define datasets, metrics, graders, and regression gates for AI systems (product features or this skill portfolio).

# When to Use / When NOT to Use

**Use when:** LLM/agent features; prompt/skill changes; “are we better?”; offline evals; skill portfolio regression.

**Do not use when:** non-AI CRUD with no model calls; pure infra without generative behavior.

# Preconditions

Clear target behavior and some way to run the system/skill under test (or design-only mode if runtime unavailable).

# Inputs / Outputs

**Inputs:** feature/skill under test, CONTEXT_PACK, success criteria, existing evals if any.

**Outputs:** `EVAL_PLAN` then `EVAL_REPORT` when executed.

# Upstream / Downstream

**Upstream:** spec-validator, test-strategy-designer, engineering-os, skill authors.

**Downstream:** feature-implementer, quality-gate, pr-generator, delivery-planner.

# Core Principles

1. Evals need fixed cases + graded outcomes — not vibes alone.
2. Split: unit-ish (deterministic checks), scenario, adversarial/safety, human review sample.
3. Measure regressions; version datasets.
4. Cost/latency are metrics too.
5. For skills: trigger accuracy, artifact validity, stop-condition obedience.
6. Report confidence intervals qualitatively when N is small.
7. Never fabricate scores.

# Process

## Modes
- `product` — AI feature in the app
- `skill` — evaluate an Agent Skill / portfolio path

1. Define behaviors and failure modes to catch.
2. Build/extend dataset cases (inputs, expected traits, forbidden traits).
3. Choose graders: exact, rubric LLM-as-judge (with caution), programmatic assertions.
4. Set pass thresholds and CI gating vs offline-only.
5. Run if possible → EVAL_REPORT with per-case results.
6. Recommend next dataset expansions.

# Evidence Requirements

Case IDs, commands, raw graded outputs summarized (redact PII). N stated.

# Stop Conditions / Failure Modes

| Condition | Action |
|-----------|--------|
| No cases defined | Block “quality” claims |
| Runtime unavailable | Ship EVAL_PLAN only; Decision ProceedWithConditions |
| Safety cases failing | Block ship for product mode |

# Severity + Confidence

Safety/eval failures on HighRisk AI → Critical/High for release.

# Output Contract

```
## EVAL_PLAN
Mode: product | skill
Behaviors: ...
Dataset: ...
Graders: ...
Thresholds: ...
## EVAL_REPORT (if run)
N: ...
Results summary: ...
Failures: ...
Decision: Proceed | ProceedWithConditions | Revise | Block
```

# Handoffs

**quality-gate**, **feature-implementer**, **test-strategy-designer**, **security-auditor** (prompt injection / data leak).

# Never

- Never invent pass rates.
- Never rely solely on a single cherry-picked demo prompt.
- Never log secrets from model outputs.