Agent Evaluation
Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks
When to Use
- User mentions or implies: agent testing
- User mentions or implies: agent evaluation
- User mentions or implies: benchmark agents
- User mentions or implies: agent reliability
- User mentions or implies: test agent
Core Workflow
- Confirm the request matches this skill's trigger, scope, and risk profile.
- Use the topic map to identify the relevant pattern, checklist, or example before writing detailed guidance or code.
- Load
references/full-guidance.md when implementation details, examples, anti-patterns, validation checks, or edge cases are needed.
- Apply only the relevant guidance instead of loading or repeating the entire reference by default.
- Verify the result against any validation checks, limitations, security notes, or platform constraints in the reference.
Topic Map
- Capabilities
- Prerequisites
- Scope
- Ecosystem
- Primary_tools
- Alternatives
- Deprecated
- Patterns
- Statistical Test Evaluation
- Behavioral Contract Testing
- Adversarial Testing
- Regression Testing Pipeline
- Sharp Edges
- Agent scores well on benchmarks but fails in production
- Same test passes sometimes, fails other times
- Agent optimized for metric, not actual task
- Test data accidentally used in training or prompts
- Delegation Triggers
Reference Map
references/full-guidance.md preserves the complete original guidance, including examples and detailed edge cases.
Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
Progressive Loading
Keep this SKILL.md as the compact routing and workflow entrypoint. Load the reference file only when the user task requires the deeper implementation material.
1---2name: agent-evaluation3description: Testing and benchmarking LLM agents including behavioral testing,4license: MIT5---67# Agent Evaluation89Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks1011## When to Use12- User mentions or implies: agent testing13- User mentions or implies: agent evaluation14- User mentions or implies: benchmark agents15- User mentions or implies: agent reliability16- User mentions or implies: test agent1718## Core Workflow191. Confirm the request matches this skill's trigger, scope, and risk profile.202. Use the topic map to identify the relevant pattern, checklist, or example before writing detailed guidance or code.213. Load `references/full-guidance.md` when implementation details, examples, anti-patterns, validation checks, or edge cases are needed.224. Apply only the relevant guidance instead of loading or repeating the entire reference by default.235. Verify the result against any validation checks, limitations, security notes, or platform constraints in the reference.2425## Topic Map26- Capabilities27- Prerequisites28- Scope29- Ecosystem30- Primary_tools31- Alternatives32- Deprecated33- Patterns34- Statistical Test Evaluation35- Behavioral Contract Testing36- Adversarial Testing37- Regression Testing Pipeline38- Sharp Edges39- Agent scores well on benchmarks but fails in production40- Same test passes sometimes, fails other times41- Agent optimized for metric, not actual task42- Test data accidentally used in training or prompts43- Delegation Triggers4445## Reference Map46- `references/full-guidance.md` preserves the complete original guidance, including examples and detailed edge cases.4748## Limitations49- Use this skill only when the task clearly matches the scope described above.50- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.51- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.5253## Progressive Loading54Keep this `SKILL.md` as the compact routing and workflow entrypoint. Load the reference file only when the user task requires the deeper implementation material.
newmindsgroup