Customization
Before executing, check for user customizations at:
~/.claude/skills/PAI/USER/SKILLCUSTOMIZATIONS/Evals/
If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.
- Output text notification:
Running the **WorkflowName** workflow in the **Evals** skill to ACTION...
This is not optional. Execute this curl command immediately upon skill invocation.
Evals - AI Agent Evaluation Framework
Comprehensive agent evaluation system based on Anthropic's "Demystifying Evals for AI Agents" (Jan 2026).
Key differentiator: Evaluates agent workflows (transcripts, tool calls, multi-turn conversations), not just single outputs.
When to Activate
- "run evals", "test this agent", "evaluate", "check quality", "benchmark"
- "regression test", "capability test"
- Compare agent behaviors across changes
- Validate agent workflows before deployment
- Verify ALGORITHM ISC rows
- Create new evaluation tasks from failures
Core Concepts
Three Grader Types
| Type |
Strengths |
Weaknesses |
Use For |
| Code-based |
Fast, cheap, deterministic, reproducible |
Brittle, lacks nuance |
Tests, state checks, tool verification |
| Model-based |
Flexible, captures nuance, scalable |
Non-deterministic, expensive |
Quality rubrics, assertions, comparisons |
| Human |
Gold standard, handles subjectivity |
Expensive, slow |
Calibration, spot checks, A/B testing |
Evaluation Types
| Type |
Pass Target |
Purpose |
| Capability |
~70% |
Stretch goals, measuring improvement potential |
| Regression |
~99% |
Quality gates, detecting backsliding |
Key Metrics
- pass@k: Probability of at least 1 success in k trials (measures capability)
- pass^k: Probability all k trials succeed (measures consistency/reliability)
Workflow Routing
| Trigger |
Workflow |
| "run evals", "evaluate suite" |
Run suite via Tools/AlgorithmBridge.ts |
| "log failure" |
Log failure via Tools/FailureToTask.ts log |
| "convert failures" |
Convert to tasks via Tools/FailureToTask.ts convert-all |
| "create suite" |
Create suite via Tools/SuiteManager.ts create |
| "check saturation" |
Check via Tools/SuiteManager.ts check-saturation |
Quick Reference
CLI Commands
# Run an eval suite
bun run ~/.claude/skills/Evals/Tools/AlgorithmBridge.ts -s <suite>
# Log a failure for later conversion
bun run ~/.claude/skills/Evals/Tools/FailureToTask.ts log "description" -c category -s severity
# Convert failures to test tasks
bun run ~/.claude/skills/Evals/Tools/FailureToTask.ts convert-all
# Manage suites
bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts create <name> -t capability -d "description"
bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts list
bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts check-saturation <name>
bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts graduate <name>
ALGORITHM Integration
Evals is a verification method for THE ALGORITHM ISC rows:
# Run eval and update ISC row
bun run ~/.claude/skills/Evals/Tools/AlgorithmBridge.ts -s regression-core -r 3 -u
ISC rows can specify eval verification:
| # | What Ideal Looks Like | Verify |
|---|----------------------|--------|
| 1 | Auth bypass fixed | eval:auth-security |
| 2 | Tests all pass | eval:regression |
Available Graders
Code-Based (Fast, Deterministic)
| Grader |
Use Case |
string_match |
Exact substring matching |
regex_match |
Pattern matching |
binary_tests |
Run test files |
static_analysis |
Lint, type-check, security scan |
state_check |
Verify system state after execution |
tool_calls |
Verify specific tools were called |
Model-Based (Nuanced)
| Grader |
Use Case |
llm_rubric |
Score against detailed rubric |
natural_language_assert |
Check assertions are true |
pairwise_comparison |
Compare to reference with position swap |
Domain Patterns
Pre-configured grader stacks for common agent types:
| Domain |
Primary Graders |
coding |
binary_tests + static_analysis + tool_calls + llm_rubric |
conversational |
llm_rubric + natural_language_assert + state_check |
research |
llm_rubric + natural_language_assert + tool_calls |
computer_use |
state_check + tool_calls + llm_rubric |
See Data/DomainPatterns.yaml for full configurations.
Task Schema (YAML)
task:
id: "fix-auth-bypass_1"
description: "Fix authentication bypass when password is empty"
type: regression # or capability
domain: coding
graders:
- type: binary_tests
required: [test_empty_pw.py]
weight: 0.30
- type: tool_calls
weight: 0.20
params:
sequence: [read_file, edit_file, run_tests]
- type: llm_rubric
weight: 0.50
params:
rubric: prompts/security_review.md
trials: 3
pass_threshold: 0.75
Resource Index
| Resource |
Purpose |
Types/index.ts |
Core type definitions |
Graders/CodeBased/ |
Deterministic graders |
Graders/ModelBased/ |
LLM-powered graders |
Tools/TranscriptCapture.ts |
Capture agent trajectories |
Tools/TrialRunner.ts |
Multi-trial execution with pass@k |
Tools/SuiteManager.ts |
Suite management and saturation |
Tools/FailureToTask.ts |
Convert failures to test tasks |
Tools/AlgorithmBridge.ts |
ALGORITHM integration |
Data/DomainPatterns.yaml |
Domain-specific grader configs |
Key Principles (from Anthropic)
- Start with 20-50 real failures - Don't overthink, capture what actually broke
- Unambiguous tasks - Two experts should reach identical verdicts
- Balanced problem sets - Test both "should do" AND "should NOT do"
- Grade outputs, not paths - Don't penalize valid creative solutions
- Calibrate LLM judges - Against human expert judgment
- Check transcripts regularly - Verify graders work correctly
- Monitor saturation - Graduate to regression when hitting 95%+
- Build infrastructure early - Evals shape how quickly you can adopt new models
Related
- ALGORITHM: Evals is a verification method
- Science: Evals implements scientific method
- Browser: For visual verification graders
1---2name: evals3description: Agent evaluation framework based on Anthropic's best practices. USE WHEN eval, evaluate, test agent, benchmark, verify behavior, regression test, capability test. Includes three grader types (code-based, model-based, human), transcript capture, pass@k/pass^k metrics, and ALGORITHM integration.4---56## Customization78**Before executing, check for user customizations at:**9`~/.claude/skills/PAI/USER/SKILLCUSTOMIZATIONS/Evals/`1011If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.12132. **Output text notification**:14 ```15 Running the **WorkflowName** workflow in the **Evals** skill to ACTION...16 ```1718**This is not optional. Execute this curl command immediately upon skill invocation.**1920# Evals - AI Agent Evaluation Framework2122Comprehensive agent evaluation system based on Anthropic's "Demystifying Evals for AI Agents" (Jan 2026).2324**Key differentiator:** Evaluates agent *workflows* (transcripts, tool calls, multi-turn conversations), not just single outputs.2526---2728## When to Activate2930- "run evals", "test this agent", "evaluate", "check quality", "benchmark"31- "regression test", "capability test"32- Compare agent behaviors across changes33- Validate agent workflows before deployment34- Verify ALGORITHM ISC rows35- Create new evaluation tasks from failures3637---3839## Core Concepts4041### Three Grader Types4243| Type | Strengths | Weaknesses | Use For |44|------|-----------|------------|---------|45| **Code-based** | Fast, cheap, deterministic, reproducible | Brittle, lacks nuance | Tests, state checks, tool verification |46| **Model-based** | Flexible, captures nuance, scalable | Non-deterministic, expensive | Quality rubrics, assertions, comparisons |47| **Human** | Gold standard, handles subjectivity | Expensive, slow | Calibration, spot checks, A/B testing |4849### Evaluation Types5051| Type | Pass Target | Purpose |52|------|-------------|---------|53| **Capability** | ~70% | Stretch goals, measuring improvement potential |54| **Regression** | ~99% | Quality gates, detecting backsliding |5556### Key Metrics5758- **pass@k**: Probability of at least 1 success in k trials (measures capability)59- **pass^k**: Probability all k trials succeed (measures consistency/reliability)6061---6263## Workflow Routing6465| Trigger | Workflow |66|---------|----------|67| "run evals", "evaluate suite" | Run suite via `Tools/AlgorithmBridge.ts` |68| "log failure" | Log failure via `Tools/FailureToTask.ts log` |69| "convert failures" | Convert to tasks via `Tools/FailureToTask.ts convert-all` |70| "create suite" | Create suite via `Tools/SuiteManager.ts create` |71| "check saturation" | Check via `Tools/SuiteManager.ts check-saturation` |7273---7475## Quick Reference7677### CLI Commands7879```bash80# Run an eval suite81bun run ~/.claude/skills/Evals/Tools/AlgorithmBridge.ts -s <suite>8283# Log a failure for later conversion84bun run ~/.claude/skills/Evals/Tools/FailureToTask.ts log "description" -c category -s severity8586# Convert failures to test tasks87bun run ~/.claude/skills/Evals/Tools/FailureToTask.ts convert-all8889# Manage suites90bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts create <name> -t capability -d "description"91bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts list92bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts check-saturation <name>93bun run ~/.claude/skills/Evals/Tools/SuiteManager.ts graduate <name>94```9596### ALGORITHM Integration9798Evals is a verification method for THE ALGORITHM ISC rows:99100```bash101# Run eval and update ISC row102bun run ~/.claude/skills/Evals/Tools/AlgorithmBridge.ts -s regression-core -r 3 -u103```104105ISC rows can specify eval verification:106```107| # | What Ideal Looks Like | Verify |108|---|----------------------|--------|109| 1 | Auth bypass fixed | eval:auth-security |110| 2 | Tests all pass | eval:regression |111```112113---114115## Available Graders116117### Code-Based (Fast, Deterministic)118119| Grader | Use Case |120|--------|----------|121| `string_match` | Exact substring matching |122| `regex_match` | Pattern matching |123| `binary_tests` | Run test files |124| `static_analysis` | Lint, type-check, security scan |125| `state_check` | Verify system state after execution |126| `tool_calls` | Verify specific tools were called |127128### Model-Based (Nuanced)129130| Grader | Use Case |131|--------|----------|132| `llm_rubric` | Score against detailed rubric |133| `natural_language_assert` | Check assertions are true |134| `pairwise_comparison` | Compare to reference with position swap |135136---137138## Domain Patterns139140Pre-configured grader stacks for common agent types:141142| Domain | Primary Graders |143|--------|-----------------|144| `coding` | binary_tests + static_analysis + tool_calls + llm_rubric |145| `conversational` | llm_rubric + natural_language_assert + state_check |146| `research` | llm_rubric + natural_language_assert + tool_calls |147| `computer_use` | state_check + tool_calls + llm_rubric |148149See `Data/DomainPatterns.yaml` for full configurations.150151---152153## Task Schema (YAML)154155```yaml156task:157 id: "fix-auth-bypass_1"158 description: "Fix authentication bypass when password is empty"159 type: regression # or capability160 domain: coding161162 graders:163 - type: binary_tests164 required: [test_empty_pw.py]165 weight: 0.30166167 - type: tool_calls168 weight: 0.20169 params:170 sequence: [read_file, edit_file, run_tests]171172 - type: llm_rubric173 weight: 0.50174 params:175 rubric: prompts/security_review.md176177 trials: 3178 pass_threshold: 0.75179```180181---182183## Resource Index184185| Resource | Purpose |186|----------|---------|187| `Types/index.ts` | Core type definitions |188| `Graders/CodeBased/` | Deterministic graders |189| `Graders/ModelBased/` | LLM-powered graders |190| `Tools/TranscriptCapture.ts` | Capture agent trajectories |191| `Tools/TrialRunner.ts` | Multi-trial execution with pass@k |192| `Tools/SuiteManager.ts` | Suite management and saturation |193| `Tools/FailureToTask.ts` | Convert failures to test tasks |194| `Tools/AlgorithmBridge.ts` | ALGORITHM integration |195| `Data/DomainPatterns.yaml` | Domain-specific grader configs |196197---198199## Key Principles (from Anthropic)2002011. **Start with 20-50 real failures** - Don't overthink, capture what actually broke2022. **Unambiguous tasks** - Two experts should reach identical verdicts2033. **Balanced problem sets** - Test both "should do" AND "should NOT do"2044. **Grade outputs, not paths** - Don't penalize valid creative solutions2055. **Calibrate LLM judges** - Against human expert judgment2066. **Check transcripts regularly** - Verify graders work correctly2077. **Monitor saturation** - Graduate to regression when hitting 95%+2088. **Build infrastructure early** - Evals shape how quickly you can adopt new models209210---211212## Related213214- **ALGORITHM**: Evals is a verification method215- **Science**: Evals implements scientific method216- **Browser**: For visual verification graders
kai-cli