How do I install the AI Eval Harness skill?

Run npx skillmds@latest add willianbs/ai-eval-harness in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is AI Eval Harness safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with AI Eval Harness?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is AI Eval Harness free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published AI Eval Harness?

willianbs (@willianbs) published this skill. Their other Agent Skills are listed on their SkillMD profile.

AI Eval Harness by willianbs — Agent Skill

Name: AI Eval Harness
Author: willianbs

AI Eval Harness

Designs and runs evaluation harnesses for AI features and for agent skills themselves. Use when shipping LLM features or measuring skill quality. Emits EVAL_PLAN / EVAL_REPORT. Never claims production quality from a handful of vibes-only prompts.

willianbs 6e00ff6 2 files · 3.7 KB Updated Sep 17, 2026

File contents

Purpose

Define datasets, metrics, graders, and regression gates for AI systems (product features or this skill portfolio).

When to Use / When NOT to Use

Use when: LLM/agent features; prompt/skill changes; “are we better?”; offline evals; skill portfolio regression.

Do not use when: non-AI CRUD with no model calls; pure infra without generative behavior.

Preconditions

Clear target behavior and some way to run the system/skill under test (or design-only mode if runtime unavailable).

Inputs / Outputs

Inputs: feature/skill under test, CONTEXT_PACK, success criteria, existing evals if any.

Outputs: EVAL_PLAN then EVAL_REPORT when executed.

Upstream / Downstream

Upstream: spec-validator, test-strategy-designer, engineering-os, skill authors.

Downstream: feature-implementer, quality-gate, pr-generator, delivery-planner.

Core Principles

Evals need fixed cases + graded outcomes — not vibes alone.
Split: unit-ish (deterministic checks), scenario, adversarial/safety, human review sample.
Measure regressions; version datasets.
Cost/latency are metrics too.
For skills: trigger accuracy, artifact validity, stop-condition obedience.
Report confidence intervals qualitatively when N is small.
Never fabricate scores.

Process

Modes

product — AI feature in the app
skill — evaluate an Agent Skill / portfolio path

Define behaviors and failure modes to catch.
Build/extend dataset cases (inputs, expected traits, forbidden traits).
Choose graders: exact, rubric LLM-as-judge (with caution), programmatic assertions.
Set pass thresholds and CI gating vs offline-only.
Run if possible → EVAL_REPORT with per-case results.
Recommend next dataset expansions.

Evidence Requirements

Case IDs, commands, raw graded outputs summarized (redact PII). N stated.

Stop Conditions / Failure Modes

Condition	Action
No cases defined	Block “quality” claims
Runtime unavailable	Ship EVAL_PLAN only; Decision ProceedWithConditions
Safety cases failing	Block ship for product mode

Severity + Confidence

Safety/eval failures on HighRisk AI → Critical/High for release.

Output Contract

## EVAL_PLAN
Mode: product | skill
Behaviors: ...
Dataset: ...
Graders: ...
Thresholds: ...
## EVAL_REPORT (if run)
N: ...
Results summary: ...
Failures: ...
Decision: Proceed | ProceedWithConditions | Revise | Block

Handoffs

quality-gate, feature-implementer, test-strategy-designer, security-auditor (prompt injection / data leak).

Never

Never invent pass rates.
Never rely solely on a single cherry-picked demo prompt.
Never log secrets from model outputs.

---name: ai-eval-harnessdescription: Designs and runs evaluation harnesses for AI features and for agent skills themselves. Use when shipping LLM features or measuring skill quality. Emits EVAL_PLAN / EVAL_REPORT. Never claims production quality from a handful of vibes-only prompts.---# PurposeDefine datasets, metrics, graders, and regression gates for AI systems (product features or this skill portfolio).# When to Use / When NOT to Use**Use when:** LLM/agent features; prompt/skill changes; “are we better?”; offline evals; skill portfolio regression.**Do not use when:** non-AI CRUD with no model calls; pure infra without generative behavior.# PreconditionsClear target behavior and some way to run the system/skill under test (or design-only mode if runtime unavailable).# Inputs / Outputs**Inputs:** feature/skill under test, CONTEXT_PACK, success criteria, existing evals if any.**Outputs:** `EVAL_PLAN` then `EVAL_REPORT` when executed.# Upstream / Downstream**Upstream:** spec-validator, test-strategy-designer, engineering-os, skill authors.**Downstream:** feature-implementer, quality-gate, pr-generator, delivery-planner.# Core Principles1. Evals need fixed cases + graded outcomes — not vibes alone.2. Split: unit-ish (deterministic checks), scenario, adversarial/safety, human review sample.3. Measure regressions; version datasets.4. Cost/latency are metrics too.5. For skills: trigger accuracy, artifact validity, stop-condition obedience.6. Report confidence intervals qualitatively when N is small.7. Never fabricate scores.# Process## Modes- `product` — AI feature in the app- `skill` — evaluate an Agent Skill / portfolio path1. Define behaviors and failure modes to catch.2. Build/extend dataset cases (inputs, expected traits, forbidden traits).3. Choose graders: exact, rubric LLM-as-judge (with caution), programmatic assertions.4. Set pass thresholds and CI gating vs offline-only.5. Run if possible → EVAL_REPORT with per-case results.6. Recommend next dataset expansions.# Evidence RequirementsCase IDs, commands, raw graded outputs summarized (redact PII). N stated.# Stop Conditions / Failure Modes| Condition | Action ||-----------|--------|| No cases defined | Block “quality” claims || Runtime unavailable | Ship EVAL_PLAN only; Decision ProceedWithConditions || Safety cases failing | Block ship for product mode |# Severity + ConfidenceSafety/eval failures on HighRisk AI → Critical/High for release.# Output Contract```## EVAL_PLANMode: product | skillBehaviors: ...Dataset: ...Graders: ...Thresholds: ...## EVAL_REPORT (if run)N: ...Results summary: ...Failures: ...Decision: Proceed | ProceedWithConditions | Revise | Block```# Handoffs**quality-gate**, **feature-implementer**, **test-strategy-designer**, **security-auditor** (prompt injection / data leak).# Never- Never invent pass rates.- Never rely solely on a single cherry-picked demo prompt.- Never log secrets from model outputs.

Frequently asked questions