How do I install the Agent Benchmark skill?

Run npx skillmds@latest add tomevault-io/agent-benchmark in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Agent Benchmark safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Agent Benchmark?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Agent Benchmark free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Agent Benchmark?

tomevault-io (@tomevault-io) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Agent Benchmark by tomevault-io — Agent Skill

Name: Agent Benchmark
Author: tomevault-io

Agent Benchmark

Expert in writing test configurations for agent-benchmark, a testing framework for AI agents using MCP servers. Use when creating YAML test files, configuring providers, servers, agents, sessions, assertions, or using templates. Helps write benchmarks for AI coding agents. Use when this capability is needed.

tomevault-io ee8b01c 2 files · 3.2 KB Updated Sep 17, 2026

File contents

Agent Benchmark Testing Expert

You are an expert in writing test configurations for agent-benchmark, a YAML-based testing framework for AI agents that interact with MCP (Model Context Protocol) servers.

Core Concepts

agent-benchmark tests AI agents by:

Connecting agents to LLM providers (OpenAI, Azure, Anthropic, Google, etc.)
Giving agents access to MCP servers (tools)
Running prompts and validating behavior with assertions

YAML Structure

Every test file has these sections:

providers:    # LLM configurations
servers:      # MCP server definitions  
agents:       # Agent configurations (provider + servers)
sessions:     # Test sessions containing tests
settings:     # Global settings
variables:    # Reusable template variables
criteria:     # Success rate requirements

Quick Start Example

providers:
  - name: gpt4
    type: AZURE
    auth_type: entra_id
    model: gpt-4o
    baseUrl: "{{AZURE_OPENAI_ENDPOINT}}"
    version: 2025-01-01-preview

servers:
  - name: filesystem
    type: stdio
    command: npx @modelcontextprotocol/server-filesystem /tmp

agents:
  - name: test-agent
    provider: gpt4
    servers:
      - name: filesystem
    system_prompt: |
      Execute tasks directly without asking for confirmation.

settings:
  verbose: true
  max_iterations: 10

sessions:
  - name: File Operations
    tests:
      - name: Create file
        prompt: "Create a file called test.txt with 'Hello World'"
        assertions:
          - type: tool_called
            tool: write_file
          - type: no_error_messages

Reference Documentation

For detailed configuration options, see:

@references/providers.md - LLM provider configuration (Azure, OpenAI, Anthropic, Google, Groq)
@references/assertions.md - All 20+ assertion types with examples
@references/templates.md - Template helpers (random values, timestamps, faker)
@references/advanced-features.md - Rate limiting, 429 retry, AI analysis, skills, clarification detection
@references/best-practices.md - Tips for reliable test configurations

Converted and distributed by TomeVault — claim your Tome and manage your conversions.

---name: agent-benchmarkdescription: Expert in writing test configurations for agent-benchmark, a testing framework for AI agents using MCP servers. Use when creating YAML test files, configuring providers, servers, agents, sessions, assertions, or using templates. Helps write benchmarks for AI coding agents. Use when this capability is needed.---# Agent Benchmark Testing ExpertYou are an expert in writing test configurations for **agent-benchmark**, a YAML-based testing framework for AI agents that interact with MCP (Model Context Protocol) servers.## Core Conceptsagent-benchmark tests AI agents by:1. Connecting agents to LLM providers (OpenAI, Azure, Anthropic, Google, etc.)2. Giving agents access to MCP servers (tools)3. Running prompts and validating behavior with assertions## YAML StructureEvery test file has these sections:```yamlproviders: # LLM configurationsservers: # MCP server definitions agents: # Agent configurations (provider + servers)sessions: # Test sessions containing testssettings: # Global settingsvariables: # Reusable template variablescriteria: # Success rate requirements```## Quick Start Example```yamlproviders: - name: gpt4 type: AZURE auth_type: entra_id model: gpt-4o baseUrl: "{{AZURE_OPENAI_ENDPOINT}}" version: 2025-01-01-previewservers: - name: filesystem type: stdio command: npx @modelcontextprotocol/server-filesystem /tmpagents: - name: test-agent provider: gpt4 servers: - name: filesystem system_prompt: | Execute tasks directly without asking for confirmation.settings: verbose: true max_iterations: 10sessions: - name: File Operations tests: - name: Create file prompt: "Create a file called test.txt with 'Hello World'" assertions: - type: tool_called tool: write_file - type: no_error_messages```## Reference DocumentationFor detailed configuration options, see:- @references/providers.md - LLM provider configuration (Azure, OpenAI, Anthropic, Google, Groq)- @references/assertions.md - All 20+ assertion types with examples- @references/templates.md - Template helpers (random values, timestamps, faker)- @references/advanced-features.md - Rate limiting, 429 retry, AI analysis, skills, clarification detection- @references/best-practices.md - Tips for reliable test configurations---> Converted and distributed by [TomeVault](https://tomevault.io/claim/mykhaliev) — claim your Tome and manage your conversions.

Frequently asked questions