MCP Agent Evaluation
Use this skill to design practical evaluations for agents that discover, select, and call MCP tools. Focus on whether the agent can solve real tasks with live or simulated toolchains, not only whether it can call a single tool correctly.
Activation Checklist
Activate when the user asks to:
- Benchmark an MCP client, coding agent, or multi-tool agent
- Compare models on tool selection, planning, or task completion
- Build evaluation tasks from real MCP servers
- Diagnose failed tool-use trajectories
- Add security or prompt-injection tests around MCP tool metadata and outputs
Evaluation Model
Evaluate at four levels:
- Tool discovery - the agent finds relevant tools without being handed exact tool names.
- Schema and parameter use - the agent fills arguments correctly, handles optional fields, and respects tool constraints.
- Trajectory quality - the agent sequences calls coherently, uses intermediate outputs, and recovers from tool errors.
- Outcome quality - the final answer is correct, grounded in tool results, and safe to deliver.
Add a fifth track for security when untrusted tool metadata, retrieved content, or tool outputs can influence future calls.
Workflow
Inventory the environment
- List MCP servers, tools, schemas, authentication boundaries, side effects, and domain coverage.
- Mark tools as read-only, write-capable, privileged, networked, or user-data-sensitive.
- Record which tools are complementary and should be used together.
Create task families
- Fuzzy lookup: user describes the goal without naming the tool.
- Multi-hop: success requires two or more dependent calls.
- Cross-domain: success requires tools from different servers.
- Error recovery: one call fails or returns incomplete data.
- Grounding: answer must cite or summarize intermediate tool outputs.
- Security: malicious descriptions, injected tool output, overbroad scopes, and unsafe write attempts.
Define scoring before running agents
- Use binary pass/fail for critical outcomes.
- Use graded rubrics for trajectory, grounding, and recovery.
- Separate model failures from harness failures and unavailable tools.
- Store full trajectories, not just final answers.
Run minimal baselines
- Single explicit tool task.
- Single server multi-step task.
- Multi-server task with fuzzy instructions.
- Security canary task where the correct behavior is refusal, confirmation, or sandboxing.
Analyze failure modes
- Wrong tool chosen
- Correct tool with wrong arguments
- Correct calls but poor state tracking
- Ungrounded final answer
- Unsafe trust in tool metadata or output
- Excessive retries or runaway tool loops
Rubric
| Dimension |
Pass Signal |
Failure Signal |
| Tool retrieval |
Relevant tools selected from vague request |
Agent ignores available tools or picks lexical near misses |
| Schema use |
Required and optional fields are valid |
Hallucinated fields, invalid enum values, missing constraints |
| Planning |
Calls are ordered by dependency |
Calls are redundant, circular, or skip prerequisite data |
| Grounding |
Final answer reflects tool outputs |
Final answer invents results or ignores returned evidence |
| Recovery |
Agent adapts to errors or ambiguity |
Agent repeats the same failed call |
| Safety |
Side effects are gated and instructions are isolated |
Tool output or metadata overrides user/system intent |
Helper Script
Use mcp_eval_plan.py to turn a server/task inventory into a Markdown evaluation plan:
python scripts/mcp_eval_plan.py inventory.json
The input may contain domains, servers, task_families, and risks. Missing fields use conservative defaults.
References
Read benchmark-map.md before designing a new benchmark suite or choosing which public MCP benchmark to mirror.
External grounding:
1---2name: mcp-agent-evaluation3description: Use when evaluating MCP-enabled agents, tool-using LLM systems, multi-server workflows, tool-selection quality, trajectory quality, task completion, or MCP security regressions.4---56# MCP Agent Evaluation78Use this skill to design practical evaluations for agents that discover, select, and call MCP tools. Focus on whether the agent can solve real tasks with live or simulated toolchains, not only whether it can call a single tool correctly.910## Activation Checklist1112Activate when the user asks to:1314- Benchmark an MCP client, coding agent, or multi-tool agent15- Compare models on tool selection, planning, or task completion16- Build evaluation tasks from real MCP servers17- Diagnose failed tool-use trajectories18- Add security or prompt-injection tests around MCP tool metadata and outputs1920## Evaluation Model2122Evaluate at four levels:23241. **Tool discovery** - the agent finds relevant tools without being handed exact tool names.252. **Schema and parameter use** - the agent fills arguments correctly, handles optional fields, and respects tool constraints.263. **Trajectory quality** - the agent sequences calls coherently, uses intermediate outputs, and recovers from tool errors.274. **Outcome quality** - the final answer is correct, grounded in tool results, and safe to deliver.2829Add a fifth track for security when untrusted tool metadata, retrieved content, or tool outputs can influence future calls.3031## Workflow32331. **Inventory the environment**34 - List MCP servers, tools, schemas, authentication boundaries, side effects, and domain coverage.35 - Mark tools as read-only, write-capable, privileged, networked, or user-data-sensitive.36 - Record which tools are complementary and should be used together.37382. **Create task families**39 - Fuzzy lookup: user describes the goal without naming the tool.40 - Multi-hop: success requires two or more dependent calls.41 - Cross-domain: success requires tools from different servers.42 - Error recovery: one call fails or returns incomplete data.43 - Grounding: answer must cite or summarize intermediate tool outputs.44 - Security: malicious descriptions, injected tool output, overbroad scopes, and unsafe write attempts.45463. **Define scoring before running agents**47 - Use binary pass/fail for critical outcomes.48 - Use graded rubrics for trajectory, grounding, and recovery.49 - Separate model failures from harness failures and unavailable tools.50 - Store full trajectories, not just final answers.51524. **Run minimal baselines**53 - Single explicit tool task.54 - Single server multi-step task.55 - Multi-server task with fuzzy instructions.56 - Security canary task where the correct behavior is refusal, confirmation, or sandboxing.57585. **Analyze failure modes**59 - Wrong tool chosen60 - Correct tool with wrong arguments61 - Correct calls but poor state tracking62 - Ungrounded final answer63 - Unsafe trust in tool metadata or output64 - Excessive retries or runaway tool loops6566## Rubric6768| Dimension | Pass Signal | Failure Signal |69| --- | --- | --- |70| Tool retrieval | Relevant tools selected from vague request | Agent ignores available tools or picks lexical near misses |71| Schema use | Required and optional fields are valid | Hallucinated fields, invalid enum values, missing constraints |72| Planning | Calls are ordered by dependency | Calls are redundant, circular, or skip prerequisite data |73| Grounding | Final answer reflects tool outputs | Final answer invents results or ignores returned evidence |74| Recovery | Agent adapts to errors or ambiguity | Agent repeats the same failed call |75| Safety | Side effects are gated and instructions are isolated | Tool output or metadata overrides user/system intent |7677## Helper Script7879Use [mcp_eval_plan.py](./scripts/mcp_eval_plan.py) to turn a server/task inventory into a Markdown evaluation plan:8081```bash82python scripts/mcp_eval_plan.py inventory.json83```8485The input may contain `domains`, `servers`, `task_families`, and `risks`. Missing fields use conservative defaults.8687## References8889Read [benchmark-map.md](./references/benchmark-map.md) before designing a new benchmark suite or choosing which public MCP benchmark to mirror.9091External grounding:9293- [MCP-Bench HuggingFace paper page](https://huggingface.co/papers/2508.20453)94- [MCP-Bench GitHub repository](https://github.com/Accenture/mcp-bench)95- [MCPEval HuggingFace paper page](https://huggingface.co/papers/2507.12806)96- [MCPEval GitHub repository](https://github.com/SalesforceAIResearch/MCPEval)97- [MCP Security Bench HuggingFace paper page](https://huggingface.co/papers/2510.15994)98
mouadja02