097 Index 4b90993d

Benchmarks

mcpbr supports a comprehensive suite of benchmarks for evaluating MCP servers and AI agent capabilities. Each benchmark targets different skills - from bug fixing and code generation to math reasoning, tool use, and security exploit generation.

Quick Start

# List all available benchmarks
mcpbr benchmarks

# Run a specific benchmark
mcpbr run -c config.yaml --benchmark humaneval -n 20

# Run default benchmark (SWE-bench Verified)
mcpbr run -c config.yaml

All Benchmarks

Benchmarks by Category

Software Engineering

Benchmarks that test an agent's ability to work with real codebases, fix bugs, and solve programming challenges.

Code Generation

Focused benchmarks for evaluating pure code generation from specifications.

Math & Reasoning

Benchmarks testing mathematical reasoning and multi-step problem solving.

Knowledge & QA

Benchmarks evaluating knowledge, truthfulness, and question answering.

Tool Use & Agents

Benchmarks specifically testing tool use, API interaction, and agentic capabilities.

ML Research

Code Understanding

Security

Comparing Benchmarks

Benchmark Abstraction

mcpbr uses a Protocol-based abstraction that makes it easy to add new benchmarks:

from mcpbr.benchmarks import Benchmark

class MyBenchmark:
    """Custom benchmark implementation."""

    name = "my-benchmark"

    def load_tasks(self, sample_size, task_ids, level):
        """Load tasks from dataset."""
        ...

    def normalize_task(self, task):
        """Convert to normalized BenchmarkTask format."""
        ...

    async def create_environment(self, task, docker_manager):
        """Create isolated Docker environment."""
        ...

    async def evaluate(self, env, task, solution):
        """Evaluate the solution."""
        ...

    def get_prebuilt_image(self, task):
        """Return pre-built image name or None."""
        ...

    def get_prompt_template(self):
        """Return agent prompt template."""
        ...

Each benchmark implements:

• load_tasks(): Load tasks from HuggingFace or other sources
• normalize_task(): Convert to common format
• create_environment(): Set up Docker container with dependencies
• evaluate(): Run benchmark-specific evaluation
• get_prebuilt_image(): Return pre-built image name if available
• get_prompt_template(): Provide task-appropriate instructions

See src/mcpbr/benchmarks/ for reference implementations.

Listing Benchmarks

Use the CLI to see available benchmarks:

$ mcpbr benchmarks

Available Benchmarks

┌────────────────┬──────────────────────────────────────────────────────┬─────────────────────────┐
│ Benchmark      │ Description                                          │ Output Type             │
├────────────────┼──────────────────────────────────────────────────────┼─────────────────────────┤
│ swe-bench      │ Software bug fixes in GitHub repositories             │ Patch (unified diff)    │
│ cybergym       │ Security vulnerability exploitation (PoC generation)  │ Exploit code            │
│ humaneval      │ Python function completion (code generation)          │ Function code           │
│ mcptoolbench   │ MCP tool use evaluation                               │ Tool call accuracy      │
│ gsm8k          │ Grade-school math reasoning                           │ Numeric answer          │
│ ...            │ ...                                                   │ ...                     │
└────────────────┴──────────────────────────────────────────────────────┴─────────────────────────┘

Use --benchmark flag with 'run' command to select a benchmark
Example: mcpbr run -c config.yaml --benchmark humaneval

Filtering Tasks

All benchmarks support task filtering to select specific subsets:

# Filter by difficulty level
filter_difficulty:
  - "easy"
  - "medium"

# Filter by category
filter_category:
  - "django"
  - "scikit-learn"

# Filter by tags
filter_tags:
  - "security"

# CLI filtering
mcpbr run -c config.yaml --filter-difficulty easy --filter-category browser

See Configuration for full filtering documentation.

Best Practices

Choosing a Benchmark

• Testing MCP servers: Start with SWE-bench or MCPToolBench++
• Quick smoke tests: Use HumanEval (164 tasks, fast)
• Math reasoning: Use GSM8K for basic or MATH for competition-level
• Security research: Use CyberGym with appropriate difficulty level
• Multi-language: Use Aider Polyglot or CoderEval

General Tips

1. Start small: Run with -n 5 before scaling up
2. Use pre-built images: Enabled by default for SWE-bench, much faster
3. Set appropriate timeouts: See individual benchmark pages for recommendations
4. Save results: Always use -o results.json to preserve data
5. Compare benchmarks: Run multiple benchmarks to get a comprehensive picture

Related Links

• Configuration Guide - Full configuration reference
• CLI Reference - All command options
• Best Practices - Tips for effective evaluation
• Architecture - How mcpbr works internally