tools-only mcpbr
pip install mcpbr && mcpbr init && mcpbr run -c mcpbr.yaml -n 1 -v
Benchmark your MCP server against real GitHub issues. One command, hard numbers.
Model Context Protocol Benchmark Runner
Stop guessing if your MCP server actually helps. Get hard numbers comparing tool-assisted vs. baseline agent performance on real GitHub issues.
What You Get
Real metrics showing whether your MCP server improves agent performance on SWE-bench tasks. No vibes, just data.
Evaluation Results
Summary
+-----------------+-----------+----------+
| Metric | MCP Agent | Baseline |
+-----------------+-----------+----------+
| Resolved | 8/25 | 5/25 |
| Resolution Rate | 32.0% | 20.0% |
+-----------------+-----------+----------+
Improvement: +60.0%
Why mcpbr?
MCP servers promise to make LLMs better at coding tasks. But how do you prove it?
mcpbr runs controlled experiments: same model, same tasks, same environment - the only variable is your MCP server. You get:
- Apples-to-apples comparison against a baseline agent
- Real GitHub issues from SWE-bench (not toy examples)
- Reproducible results via Docker containers with pinned dependencies
Quick Start
Option 1: Use Example Configurations (Fastest)
Get started in seconds with our ready-to-use examples:
# Set your API key
export ANTHROPIC_API_KEY="your-api-key"
# Run your first evaluation
mcpbr run -c examples/quick-start/getting-started.yaml -v
This runs 5 SWE-bench tasks with the filesystem server. Expected runtime: 15-30 minutes, cost: $2-5.
!!! tip "Explore 25+ Example Configurations"
Browse the examples/ directory for:
- **Quick Start**: Getting started, testing servers, comparing models
- **Benchmarks**: SWE-bench Lite/Full, CyberGym basic/advanced
- **MCP Servers**: Filesystem, GitHub, Brave Search, databases, custom servers
- **Scenarios**: Cost-optimized, performance-optimized, CI/CD, regression detection
See the [Examples README](https://github.com/greynewell/mcpbr/tree/main/examples/README.md) for the complete guide.
Option 2: Generate Custom Configuration
1. Set your API key
export ANTHROPIC_API_KEY="your-api-key"
2. Generate a configuration file
mcpbr init
3. Edit the configuration
Point it to your MCP server:
mcp_server:
command: "npx"
args:
- "-y"
- "@modelcontextprotocol/server-filesystem"
- "{workdir}"
env: {}
provider: "anthropic"
agent_harness: "claude-code"
model: "sonnet"
dataset: "SWE-bench/SWE-bench_Lite"
sample_size: 10
timeout_seconds: 300
max_concurrent: 4
4. Run the evaluation
mcpbr run --config mcpbr.yaml
How It Works
mcpbr runs two parallel evaluations for each SWE-bench task:
- MCP Agent: LLM with access to tools from your MCP server
- Baseline Agent: Same LLM without MCP tools
By comparing resolution rates, you can measure the effectiveness of your MCP server for code exploration and bug fixing.
Host Machine
+-----------------------------------------------------------+
| mcpbr Harness (Python) |
| - Loads SWE-bench tasks from HuggingFace |
| - Pulls pre-built Docker images |
| - Orchestrates agent runs |
| - Collects results and generates reports |
+----------------------------+------------------------------+
| docker exec
+----------------------------v------------------------------+
| Docker Container (per task) |
| - Repository at correct commit |
| - All dependencies pre-installed |
| - Claude Code CLI runs inside container |
| - Generates patches and runs tests |
+-----------------------------------------------------------+
Next Steps
- Examples - Browse 25+ example configurations
- FAQ - Frequently asked questions
- Best Practices - Guide to effective mcpbr usage
- Installation - Prerequisites and installation options
- Configuration - Full configuration reference
- CLI Reference - All available commands and options
- MCP Integration - How to test your MCP server
- Benchmarks - Available benchmarks and how to use them
- Evaluation Results - Understanding output formats
- Troubleshooting - Common issues and solutions
Built by Grey Newell