tools-only Installation
pip install mcpbr && mcpbr init && mcpbr run -c mcpbr.yaml -n 1 -v
That's it. For the full setup guide, read on.
Prerequisites
Before installing mcpbr, ensure you have the following:
| Requirement | Version | Notes |
|---|---|---|
| Python | 3.11+ | Required for mcpbr |
| Docker | Latest | Must be running |
| Claude Code CLI | Latest | The claude command |
| Network access | - | For Docker images and API calls |
API Key
You'll need an Anthropic API key:
export ANTHROPIC_API_KEY="sk-ant-..."
!!! tip "Get an API key" Sign up at console.anthropic.com to get your API key.
Claude Code CLI
Install the Claude Code CLI globally:
npm install -g @anthropic-ai/claude-code
Verify the installation:
which claude # Should return the path to the CLI
Docker
Ensure Docker is installed and running:
docker info
If Docker isn't running, start it from your system's application launcher or:
=== "macOS"
bash open -a Docker
=== "Linux"
bash sudo systemctl start docker
=== "Windows" Start Docker Desktop from the Start menu.
Installation Methods
From PyPI (Recommended)
pip install mcpbr
From npm
mcpbr is also available as an npm package for easy integration with Node.js workflows:
# Run with npx (no installation)
npx mcpbr-cli run -c config.yaml
# Or install globally
npm install -g mcpbr-cli
mcpbr run -c config.yaml
!!! info "Package Details"
Package name: mcpbr-cli
The npm package is a wrapper that requires Python 3.11+ and the mcpbr Python package to be installed separately:
```bash
pip install mcpbr
npm install -g mcpbr-cli
```
From Source
git clone https://github.com/greynewell/mcpbr.git
cd mcpbr
pip install -e .
With uv
uv pip install mcpbr
Or from source:
git clone https://github.com/greynewell/mcpbr.git
cd mcpbr
uv pip install -e .
Verify Installation
After installation, verify everything is working:
# Check mcpbr is installed
mcpbr --version
# List supported models
mcpbr models
# Generate a test config
mcpbr init -o test-config.yaml
Supported Models
mcpbr supports the following Claude models:
| Model | ID | Context Window |
|---|---|---|
| Claude Opus 4.5 | opus or claude-opus-4-5-20251101 |
200,000 |
| Claude Sonnet 4.5 | sonnet or claude-sonnet-4-5-20250929 |
200,000 |
| Claude Haiku 4.5 | haiku or claude-haiku-4-5-20251001 |
200,000 |
| Claude Opus 4 | claude-opus-4-20250514 |
200,000 |
| Claude Sonnet 4 | claude-sonnet-4-20250514 |
200,000 |
| Claude Haiku 4 | claude-haiku-4-20250514 |
200,000 |
| Claude 3.5 Sonnet | claude-3-5-sonnet-20241022 |
200,000 |
Run mcpbr models to see the full list.
Apple Silicon Notes
On ARM64 Macs (M1/M2/M3), x86_64 Docker images run via emulation. This is:
- Slower than native ARM64 images
- Required for compatibility with all SWE-bench tasks
- Automatic - no configuration needed
If you experience issues, ensure Rosetta 2 is installed:
softwareupdate --install-rosetta
Development Installation
For contributing to mcpbr:
git clone https://github.com/greynewell/mcpbr.git
cd mcpbr
pip install -e ".[dev]"
This installs additional development dependencies:
pytest- Testing frameworkpytest-asyncio- Async test supportruff- Linting and formatting
See Contributing for more details.
Next Steps
After installation, you have two options to get started:
Option 1: Use Example Configurations (Fastest)
Jump straight in with our ready-to-use examples:
# Set your API key
export ANTHROPIC_API_KEY="your-api-key"
# Run your first evaluation
mcpbr run -c examples/quick-start/getting-started.yaml -v
Explore 25+ example configurations in the examples/ directory covering benchmarks, MCP servers, and common scenarios. See the Examples README for the complete guide.
Option 2: Generate Custom Configuration
Create your own configuration:
mcpbr init
# Edit mcpbr.yaml
mcpbr run -c mcpbr.yaml -v
Continue Learning:
- Quick Start - Run your first evaluation
- Configuration - Configure your MCP server
- Examples - Browse example configurations
- CLI Reference - Explore all commands