How do I install the Evals skill?

Run npx skillmds@latest add rooseveltadvisors-claude-agent-stack/evals in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Evals skill do?

Objective eval metrics via code/model/human graders with pass@k/pass^k scoring. USE WHEN eval, evaluate, test agent, benchmark, verify behavior, regression test, capability test, run eval, compare models, compare prompts, create judge, create use case, view results, failure to task, suite manager, transcript capture, trial runner. It is listed under AI & ML on SkillMD.

Is Evals safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: executes scripts, makes network calls, reads secrets. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Evals?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Evals free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

RooseveltAdvisors (@rooseveltadvisors-claude-agent-stack) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Evals by rooseveltadvisors-claude-agent-stack

Name: Evals
Rating: 4.2 (5 reviews)
Author: RooseveltAdvisors

Customization

Before executing, check for user customizations at: ~/.claude/PAI/USER/SKILLCUSTOMIZATIONS/Evals/

If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.

🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

You MUST send this notification BEFORE doing anything else when this skill is invoked.

Send voice notification:

curl -s -X POST http://localhost:8888/notify \
  -H "Content-Type: application/json" \
  -d '{"message": "Running the WORKFLOWNAME workflow in the Evals skill to ACTION"}' \
  > /dev/null 2>&1 &

Output text notification:

Running the **WorkflowName** workflow in the **Evals** skill to ACTION...

This is not optional. Execute this curl command immediately upon skill invocation.

Evals - AI Agent Evaluation Framework

Comprehensive agent evaluation system based on Anthropic's "Demystifying Evals for AI Agents" (Jan 2026).

Key differentiator: Evaluates agent workflows (transcripts, tool calls, multi-turn conversations), not just single outputs.

When to Activate

"run evals", "test this agent", "evaluate", "check quality", "benchmark"
"regression test", "capability test"
Compare agent behaviors across changes
Validate agent workflows before deployment
Verify ALGORITHM ISC rows
Create new evaluation tasks from failures

Core Concepts

Three Grader Types

Type	Strengths	Weaknesses	Use For
Code-based	Fast, cheap, deterministic, reproducible	Brittle, lacks nuance	Tests, state checks, tool verification
Model-based	Flexible, captures nuance, scalable	Non-deterministic, expensive	Quality rubrics, assertions, comparisons
Human	Gold standard, handles subjectivity	Expensive, slow	Calibration, spot checks, A/B testing

Evaluation Types

Type	Pass Target	Purpose
Capability	~70%	Stretch goals, measuring improvement potential
Regression	~99%	Quality gates, detecting backsliding

Key Metrics

pass@k: Probability of at least 1 success in k trials (measures capability)
pass^k: Probability all k trials succeed (measures consistency/reliability)

Workflow Routing

Request Pattern	Route To
Run eval, evaluate suite, run tests, benchmark	`Workflows/RunEval.md`
Compare models, model comparison, A/B test models	`Workflows/CompareModels.md`
Compare prompts, prompt comparison, test prompts	`Workflows/ComparePrompts.md`
Create judge, model grader, evaluation judge	`Workflows/CreateJudge.md`
Create use case, new eval, test case, create suite	`Workflows/CreateUseCase.md`
View results, eval results, scores, pass rate	`Workflows/ViewResults.md`

CLI Quick Reference

Trigger	Tool
Run suite	`Tools/AlgorithmBridge.ts`
Log failure	`Tools/FailureToTask.ts log`
Convert failures	`Tools/FailureToTask.ts convert-all`
Create suite	`Tools/SuiteManager.ts create`
Check saturation	`Tools/SuiteManager.ts check-saturation`

Quick Reference

CLI Commands

# Run an eval suite
bun run ~/.claude/skills/Utilities/Evals/Tools/AlgorithmBridge.ts -s <suite>

# Log a failure for later conversion
bun run ~/.claude/skills/Utilities/Evals/Tools/FailureToTask.ts log "description" -c category -s severity

# Convert failures to test tasks
bun run ~/.claude/skills/Utilities/Evals/Tools/FailureToTask.ts convert-all

# Manage suites
bun run ~/.claude/skills/Utilities/Evals/Tools/SuiteManager.ts create <name> -t capability -d "description"
bun run ~/.claude/skills/Utilities/Evals/Tools/SuiteManager.ts list
bun run ~/.claude/skills/Utilities/Evals/Tools/SuiteManager.ts check-saturation <name>
bun run ~/.claude/skills/Utilities/Evals/Tools/SuiteManager.ts graduate <name>

ALGORITHM Integration

Evals is a verification method for THE ALGORITHM ISC rows:

# Run eval and update ISC row
bun run ~/.claude/skills/Utilities/Evals/Tools/AlgorithmBridge.ts -s regression-core -r 3 -u

ISC rows can specify eval verification:

| # | What Ideal Looks Like | Verify |
|---|----------------------|--------|
| 1 | Auth bypass fixed | eval:auth-security |
| 2 | Tests all pass | eval:regression |

Available Graders

Code-Based (Fast, Deterministic)

Grader	Use Case
`string_match`	Exact substring matching
`regex_match`	Pattern matching
`binary_tests`	Run test files
`static_analysis`	Lint, type-check, security scan
`state_check`	Verify system state after execution
`tool_calls`	Verify specific tools were called

Model-Based (Nuanced)

Grader	Use Case
`llm_rubric`	Score against detailed rubric
`natural_language_assert`	Check assertions are true
`pairwise_comparison`	Compare to reference with position swap

Domain Patterns

Pre-configured grader stacks for common agent types:

Domain	Primary Graders
`coding`	binary_tests + static_analysis + tool_calls + llm_rubric
`conversational`	llm_rubric + natural_language_assert + state_check
`research`	llm_rubric + natural_language_assert + tool_calls
`computer_use`	state_check + tool_calls + llm_rubric

See Data/DomainPatterns.yaml for full configurations.

Task Schema (YAML)

task:
  id: "fix-auth-bypass_1"
  description: "Fix authentication bypass when password is empty"
  type: regression  # or capability
  domain: coding

  graders:
    - type: binary_tests
      required: [test_empty_pw.py]
      weight: 0.30

    - type: tool_calls
      weight: 0.20
      params:
        sequence: [read_file, edit_file, run_tests]

    - type: llm_rubric
      weight: 0.50
      params:
        rubric: prompts/security_review.md

  trials: 3
  pass_threshold: 0.75

Resource Index

Resource	Purpose
`Types/index.ts`	Core type definitions
`Graders/CodeBased/`	Deterministic graders
`Graders/ModelBased/`	LLM-powered graders
`Tools/TranscriptCapture.ts`	Capture agent trajectories
`Tools/TrialRunner.ts`	Multi-trial execution with pass@k
`Tools/SuiteManager.ts`	Suite management and saturation
`Tools/FailureToTask.ts`	Convert failures to test tasks
`Tools/AlgorithmBridge.ts`	ALGORITHM integration
`Data/DomainPatterns.yaml`	Domain-specific grader configs

Key Principles (from Anthropic)

Start with 20-50 real failures - Don't overthink, capture what actually broke
Unambiguous tasks - Two experts should reach identical verdicts
Balanced problem sets - Test both "should do" AND "should NOT do"
Grade outputs, not paths - Don't penalize valid creative solutions
Calibrate LLM judges - Against human expert judgment
Check transcripts regularly - Verify graders work correctly
Monitor saturation - Graduate to regression when hitting 95%+
Build infrastructure early - Evals shape how quickly you can adopt new models

ALGORITHM: Evals is a verification method
Science: Evals implements scientific method
Browser: For visual verification graders

Evals

File contents

Customization

🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

Evals - AI Agent Evaluation Framework

When to Activate

Core Concepts

Three Grader Types

Evaluation Types

Key Metrics

Workflow Routing

CLI Quick Reference

Quick Reference

CLI Commands

ALGORITHM Integration

Available Graders

Code-Based (Fast, Deterministic)

Model-Based (Nuanced)

Domain Patterns

Task Schema (YAML)

Resource Index

Key Principles (from Anthropic)

Related

Frequently asked questions

Evals

File contents

Customization

🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

Evals - AI Agent Evaluation Framework

When to Activate

Core Concepts

Three Grader Types

Evaluation Types

Key Metrics

Workflow Routing

CLI Quick Reference

Quick Reference

CLI Commands

ALGORITHM Integration

Available Graders

Code-Based (Fast, Deterministic)

Model-Based (Nuanced)

Domain Patterns

Task Schema (YAML)

Resource Index

Key Principles (from Anthropic)

Related

Related skills

Frequently asked questions

How do I install the Evals skill?

What does the Evals skill do?

Is Evals safe to use?

Which AI agents work with Evals?

Is Evals free to use?

Who published Evals?