Customization
Before executing, check for user customizations at:
${PAI_USER_DIR}/SKILLCUSTOMIZATIONS/Evals/
If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.
- Output text notification:
Running the **WorkflowName** workflow in the **Evals** skill to ACTION...
This is not optional. Execute this curl command immediately upon skill invocation.
Evals - AI Agent Evaluation Framework
Comprehensive agent evaluation system based on Anthropic's "Demystifying Evals for AI Agents" (Jan 2026).
Key differentiator: Evaluates agent workflows (transcripts, tool calls, multi-turn conversations), not just single outputs.
When to Activate
- "run evals", "test this agent", "evaluate", "check quality", "benchmark"
- "regression test", "capability test"
- Compare agent behaviors across changes
- Validate agent workflows before deployment
- Verify ALGORITHM ISC rows
- Create new evaluation tasks from failures
Core Concepts
Three Grader Types
| Type |
Strengths |
Weaknesses |
Use For |
| Code-based |
Fast, cheap, deterministic, reproducible |
Brittle, lacks nuance |
Tests, state checks, tool verification |
| Model-based |
Flexible, captures nuance, scalable |
Non-deterministic, expensive |
Quality rubrics, assertions, comparisons |
| Human |
Gold standard, handles subjectivity |
Expensive, slow |
Calibration, spot checks, A/B testing |
Evaluation Types
| Type |
Pass Target |
Purpose |
| Capability |
~70% |
Stretch goals, measuring improvement potential |
| Regression |
~99% |
Quality gates, detecting backsliding |
Key Metrics
- pass@k: Probability of at least 1 success in k trials (measures capability)
- pass^k: Probability all k trials succeed (measures consistency/reliability)
Workflow Routing
| Request Pattern |
Route To |
| Run eval, evaluate suite, run tests, benchmark |
Workflows/RunEval.md |
| Compare models, model comparison, A/B test models |
Workflows/CompareModels.md |
| Compare prompts, prompt comparison, test prompts |
Workflows/ComparePrompts.md |
| Create judge, model grader, evaluation judge |
Workflows/CreateJudge.md |
| Create use case, new eval, test case, create suite |
Workflows/CreateUseCase.md |
| View results, eval results, scores, pass rate |
Workflows/ViewResults.md |
| Skillgrade, skill eval, skill test, init eval, docker eval, external eval |
Workflows/Skillgrade.md |
CLI Quick Reference
| Trigger |
Tool |
| Run suite |
Tools/AlgorithmBridge.ts |
| Log failure |
Tools/FailureToTask.ts log |
| Convert failures |
Tools/FailureToTask.ts convert-all |
| Create suite |
Tools/SuiteManager.ts create |
| Check saturation |
Tools/SuiteManager.ts check-saturation |
Quick Reference
CLI Commands
# Run an eval suite
bun run ${CLAUDE_SKILL_DIR}/Tools/AlgorithmBridge.ts -s <suite>
# Log a failure for later conversion
bun run ${CLAUDE_SKILL_DIR}/Tools/FailureToTask.ts log "description" -c category -s severity
# Convert failures to test tasks
bun run ${CLAUDE_SKILL_DIR}/Tools/FailureToTask.ts convert-all
# Manage suites
bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts create <name> -t capability -d "description"
bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts list
bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts check-saturation <name>
bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts graduate <name>
ALGORITHM Integration
Evals is a verification method for THE ALGORITHM ISC rows:
# Run eval and update ISC row
bun run ${CLAUDE_SKILL_DIR}/Tools/AlgorithmBridge.ts -s regression-core -r 3 -u
ISC rows can specify eval verification:
| # | What Ideal Looks Like | Verify |
|---|----------------------|--------|
| 1 | Auth bypass fixed | eval:auth-security |
| 2 | Tests all pass | eval:regression |
Available Graders
Code-Based (Fast, Deterministic)
| Grader |
Use Case |
string_match |
Exact substring matching |
regex_match |
Pattern matching |
binary_tests |
Run test files |
static_analysis |
Lint, type-check, security scan |
state_check |
Verify system state after execution |
tool_calls |
Verify specific tools were called |
Model-Based (Nuanced)
| Grader |
Use Case |
llm_rubric |
Score against detailed rubric |
natural_language_assert |
Check assertions are true |
pairwise_comparison |
Compare to reference with position swap |
Domain Patterns
Pre-configured grader stacks for common agent types:
| Domain |
Primary Graders |
coding |
binary_tests + static_analysis + tool_calls + llm_rubric |
conversational |
llm_rubric + natural_language_assert + state_check |
research |
llm_rubric + natural_language_assert + tool_calls |
computer_use |
state_check + tool_calls + llm_rubric |
See Data/DomainPatterns.yaml for full configurations.
Task Schema (YAML)
task:
id: "fix-auth-bypass_1"
description: "Fix authentication bypass when password is empty"
type: regression # or capability
domain: coding
graders:
- type: binary_tests
required: [test_empty_pw.py]
weight: 0.30
- type: tool_calls
weight: 0.20
params:
sequence: [read_file, edit_file, run_tests]
- type: llm_rubric
weight: 0.50
params:
rubric: prompts/security_review.md
trials: 3
pass_threshold: 0.75
Resource Index
| Resource |
Purpose |
Types/index.ts |
Core type definitions |
Graders/CodeBased/ |
Deterministic graders |
Graders/ModelBased/ |
LLM-powered graders |
Tools/TranscriptCapture.ts |
Capture agent trajectories |
Tools/TrialRunner.ts |
Multi-trial execution with pass@k |
Tools/SuiteManager.ts |
Suite management and saturation |
Tools/FailureToTask.ts |
Convert failures to test tasks |
Tools/AlgorithmBridge.ts |
ALGORITHM integration |
Data/DomainPatterns.yaml |
Domain-specific grader configs |
Key Principles (from Anthropic)
- Start with 20-50 real failures - Don't overthink, capture what actually broke
- Unambiguous tasks - Two experts should reach identical verdicts
- Balanced problem sets - Test both "should do" AND "should NOT do"
- Grade outputs, not paths - Don't penalize valid creative solutions
- Calibrate LLM judges - Against human expert judgment
- Check transcripts regularly - Verify graders work correctly
- Monitor saturation - Graduate to regression when hitting 95%+
- Build infrastructure early - Evals shape how quickly you can adopt new models
Related
- ALGORITHM: Evals is a verification method
- Science: Evals implements scientific method
- Browser: For visual verification graders
1---2name: evals3description: Objective eval metrics via code/model/human graders with pass@k/pass^k scoring. USE WHEN eval, evaluate, test agent, benchmark, verify behavior, regression test, capability test, run eval, compare models, compare prompts, create judge, create use case, view results, failure to task, suite manager, transcript capture, trial runner, skillgrade, skill eval, skill test, docker eval, external eval.4---56## Customization78**Before executing, check for user customizations at:**9`${PAI_USER_DIR}/SKILLCUSTOMIZATIONS/Evals/`1011If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.121314<!-- ## 🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)1516**You MUST send this notification BEFORE doing anything else when this skill is invoked.**17181. **Send voice notification**:19 ```bash20 curl -s -X POST http://localhost:8888/notify \21 -H "Content-Type: application/json" \22 -d '{"message": "Running the WORKFLOWNAME workflow in the Evals skill to ACTION"}' \23 > /dev/null 2>&1 &24 ```25-->26272. **Output text notification**:28 ```29 Running the **WorkflowName** workflow in the **Evals** skill to ACTION...30 ```3132**This is not optional. Execute this curl command immediately upon skill invocation.**3334# Evals - AI Agent Evaluation Framework3536Comprehensive agent evaluation system based on Anthropic's "Demystifying Evals for AI Agents" (Jan 2026).3738**Key differentiator:** Evaluates agent *workflows* (transcripts, tool calls, multi-turn conversations), not just single outputs.3940---4142## When to Activate4344- "run evals", "test this agent", "evaluate", "check quality", "benchmark"45- "regression test", "capability test"46- Compare agent behaviors across changes47- Validate agent workflows before deployment48- Verify ALGORITHM ISC rows49- Create new evaluation tasks from failures5051---5253## Core Concepts5455### Three Grader Types5657| Type | Strengths | Weaknesses | Use For |58|------|-----------|------------|---------|59| **Code-based** | Fast, cheap, deterministic, reproducible | Brittle, lacks nuance | Tests, state checks, tool verification |60| **Model-based** | Flexible, captures nuance, scalable | Non-deterministic, expensive | Quality rubrics, assertions, comparisons |61| **Human** | Gold standard, handles subjectivity | Expensive, slow | Calibration, spot checks, A/B testing |6263### Evaluation Types6465| Type | Pass Target | Purpose |66|------|-------------|---------|67| **Capability** | ~70% | Stretch goals, measuring improvement potential |68| **Regression** | ~99% | Quality gates, detecting backsliding |6970### Key Metrics7172- **pass@k**: Probability of at least 1 success in k trials (measures capability)73- **pass^k**: Probability all k trials succeed (measures consistency/reliability)7475---7677## Workflow Routing7879| Request Pattern | Route To |80|---|---|81| Run eval, evaluate suite, run tests, benchmark | `Workflows/RunEval.md` |82| Compare models, model comparison, A/B test models | `Workflows/CompareModels.md` |83| Compare prompts, prompt comparison, test prompts | `Workflows/ComparePrompts.md` |84| Create judge, model grader, evaluation judge | `Workflows/CreateJudge.md` |85| Create use case, new eval, test case, create suite | `Workflows/CreateUseCase.md` |86| View results, eval results, scores, pass rate | `Workflows/ViewResults.md` |87| Skillgrade, skill eval, skill test, init eval, docker eval, external eval | `Workflows/Skillgrade.md` |8889### CLI Quick Reference9091| Trigger | Tool |92|---------|------|93| Run suite | `Tools/AlgorithmBridge.ts` |94| Log failure | `Tools/FailureToTask.ts log` |95| Convert failures | `Tools/FailureToTask.ts convert-all` |96| Create suite | `Tools/SuiteManager.ts create` |97| Check saturation | `Tools/SuiteManager.ts check-saturation` |9899---100101## Quick Reference102103### CLI Commands104105```bash106# Run an eval suite107bun run ${CLAUDE_SKILL_DIR}/Tools/AlgorithmBridge.ts -s <suite>108109# Log a failure for later conversion110bun run ${CLAUDE_SKILL_DIR}/Tools/FailureToTask.ts log "description" -c category -s severity111112# Convert failures to test tasks113bun run ${CLAUDE_SKILL_DIR}/Tools/FailureToTask.ts convert-all114115# Manage suites116bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts create <name> -t capability -d "description"117bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts list118bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts check-saturation <name>119bun run ${CLAUDE_SKILL_DIR}/Tools/SuiteManager.ts graduate <name>120```121122### ALGORITHM Integration123124Evals is a verification method for THE ALGORITHM ISC rows:125126```bash127# Run eval and update ISC row128bun run ${CLAUDE_SKILL_DIR}/Tools/AlgorithmBridge.ts -s regression-core -r 3 -u129```130131ISC rows can specify eval verification:132```133| # | What Ideal Looks Like | Verify |134|---|----------------------|--------|135| 1 | Auth bypass fixed | eval:auth-security |136| 2 | Tests all pass | eval:regression |137```138139---140141## Available Graders142143### Code-Based (Fast, Deterministic)144145| Grader | Use Case |146|--------|----------|147| `string_match` | Exact substring matching |148| `regex_match` | Pattern matching |149| `binary_tests` | Run test files |150| `static_analysis` | Lint, type-check, security scan |151| `state_check` | Verify system state after execution |152| `tool_calls` | Verify specific tools were called |153154### Model-Based (Nuanced)155156| Grader | Use Case |157|--------|----------|158| `llm_rubric` | Score against detailed rubric |159| `natural_language_assert` | Check assertions are true |160| `pairwise_comparison` | Compare to reference with position swap |161162---163164## Domain Patterns165166Pre-configured grader stacks for common agent types:167168| Domain | Primary Graders |169|--------|-----------------|170| `coding` | binary_tests + static_analysis + tool_calls + llm_rubric |171| `conversational` | llm_rubric + natural_language_assert + state_check |172| `research` | llm_rubric + natural_language_assert + tool_calls |173| `computer_use` | state_check + tool_calls + llm_rubric |174175See `Data/DomainPatterns.yaml` for full configurations.176177---178179## Task Schema (YAML)180181```yaml182task:183 id: "fix-auth-bypass_1"184 description: "Fix authentication bypass when password is empty"185 type: regression # or capability186 domain: coding187188 graders:189 - type: binary_tests190 required: [test_empty_pw.py]191 weight: 0.30192193 - type: tool_calls194 weight: 0.20195 params:196 sequence: [read_file, edit_file, run_tests]197198 - type: llm_rubric199 weight: 0.50200 params:201 rubric: prompts/security_review.md202203 trials: 3204 pass_threshold: 0.75205```206207---208209## Resource Index210211| Resource | Purpose |212|----------|---------|213| `Types/index.ts` | Core type definitions |214| `Graders/CodeBased/` | Deterministic graders |215| `Graders/ModelBased/` | LLM-powered graders |216| `Tools/TranscriptCapture.ts` | Capture agent trajectories |217| `Tools/TrialRunner.ts` | Multi-trial execution with pass@k |218| `Tools/SuiteManager.ts` | Suite management and saturation |219| `Tools/FailureToTask.ts` | Convert failures to test tasks |220| `Tools/AlgorithmBridge.ts` | ALGORITHM integration |221| `Data/DomainPatterns.yaml` | Domain-specific grader configs |222223---224225## Key Principles (from Anthropic)2262271. **Start with 20-50 real failures** - Don't overthink, capture what actually broke2282. **Unambiguous tasks** - Two experts should reach identical verdicts2293. **Balanced problem sets** - Test both "should do" AND "should NOT do"2304. **Grade outputs, not paths** - Don't penalize valid creative solutions2315. **Calibrate LLM judges** - Against human expert judgment2326. **Check transcripts regularly** - Verify graders work correctly2337. **Monitor saturation** - Graduate to regression when hitting 95%+2348. **Build infrastructure early** - Evals shape how quickly you can adopt new models235236---237238## Related239240- **ALGORITHM**: Evals is a verification method241- **Science**: Evals implements scientific method242- **Browser**: For visual verification graders
mj-deving