556 MCP Integration 8bb4aea3

MCP Server Integration

This guide explains how to benchmark your MCP (Model Context Protocol) server with mcpbr.

What is MCP?

The Model Context Protocol is an open standard that allows AI models to access external tools and data sources. MCP servers expose tools that Claude can use during agent runs.

How mcpbr Uses MCP

mcpbr runs two parallel evaluations for each task:

1. MCP Agent: Claude Code CLI with your MCP server registered
2. Baseline Agent: Claude Code CLI without MCP tools

By comparing resolution rates, you can measure the effectiveness of your MCP server.

Configuring Your MCP Server

Basic Configuration

mcp_server:
  name: "mcpbr"
  command: "npx"
  args:
    - "-y"
    - "@modelcontextprotocol/server-filesystem"
    - "{workdir}"
  env: {}

Configuration Fields

The {workdir} Placeholder

The {workdir} placeholder is replaced at runtime with /workspace - the path to the task repository inside the Docker container.

Example Configurations

Anthropic Filesystem Server

Basic file system access:

mcp_server:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]

This provides tools for reading, writing, and listing files.

Custom Python MCP Server

mcp_server:
  command: "python"
  args: ["-m", "my_mcp_server", "--workspace", "{workdir}"]
  env:
    LOG_LEVEL: "debug"
    CUSTOM_SETTING: "value"

Node.js MCP Server

mcp_server:
  command: "node"
  args: ["/path/to/server/index.js", "--root", "{workdir}"]

External API Server

mcp_server:
  command: "npx"
  args: ["-y", "@supermodeltools/mcp-server"]
  env:
    SUPERMODEL_API_KEY: "${SUPERMODEL_API_KEY}"

Testing Your Server

1. Verify Standalone

Before running mcpbr, test your MCP server independently:

# For the filesystem server
npx -y @modelcontextprotocol/server-filesystem /tmp/test

# For a custom server
python -m my_mcp_server --workspace /tmp/test

2. Quick Smoke Test

Run a single task with verbose output:

mcpbr run -c config.yaml -n 1 -v -M

This runs:

• Only 1 task (-n 1)
• With verbose output (-v)
• MCP agent only (-M)

3. Check Tool Registration

In verbose output, you'll see tool calls like:

14:23:22 astropy-12907:mcp    > mcp__mcpbr__read_file
14:23:22 astropy-12907:mcp    < File content: ...

If your tools aren't appearing, check:

• Server startup logs (stderr)
• Environment variables are set correctly
• The {workdir} path is valid

MCP Tools in Action

When your MCP server is registered, Claude can use its tools alongside built-in tools:

Built-in tools:
  Bash, Glob, Grep, Read, Write, Edit, TodoWrite, Task

MCP tools (with mcp__ prefix):
  mcp__mcpbr__read_file
  mcp__mcpbr__write_file
  mcp__mcpbr__list_directory
  mcp__mcpbr__search_files
  ...

Evaluation Strategy

Small-Scale Testing

Start with a small sample to verify your server works:

sample_size: 5
max_concurrent: 1
timeout_seconds: 180

Full Benchmark

For comprehensive results:

sample_size: null  # Full SWE-bench Lite (300 tasks)
max_concurrent: 4
timeout_seconds: 600
max_iterations: 30

Comparing Servers

To compare multiple MCP servers:

1. Create separate config files
2. Run evaluations with different configs
3. Compare JSON results

mcpbr run -c server-a.yaml -o results-a.json
mcpbr run -c server-b.yaml -o results-b.json

Server Development Tips

1. Optimize for Code Search

Tools that help locate relevant code quickly tend to improve performance:

• Semantic search
• Symbol lookup
• Reference finding

2. Provide Context

Tools that provide contextual information help Claude understand the codebase:

• File summaries
• Module structure
• Dependency graphs

3. Minimize Latency

Each tool call adds time. Consider:

• Caching results
• Batch operations
• Precomputing indexes

4. Handle Errors Gracefully

Return informative error messages that help Claude recover:

# Bad
raise Exception("Error")

# Good
raise Exception("File not found: {path}. Did you mean {suggestion}?")

Debugging

Logs Per Instance

Enable per-instance logs to debug specific tasks:

mcpbr run -c config.yaml -v --log-dir logs/

This creates JSON log files with full tool call traces.

Check Tool Usage

The results JSON includes tool usage statistics:

{
  "tool_usage": {
    "mcp__mcpbr__read_file": 15,
    "mcp__mcpbr__search_files": 8,
    "Bash": 27,
    "Read": 22
  }
}

Low MCP tool usage may indicate:

• Tools not helpful for the task
• Tool discovery issues
• Better built-in alternatives

Common Issues

Server Not Starting

Warning: MCP server add failed (exit 1): ...

Check:

• Command exists and is executable
• Environment variables are set
• No syntax errors in server code

Tools Not Appearing

If Claude isn't using your MCP tools:

• Verify server registers tools correctly
• Check tool descriptions are clear
• Ensure {workdir} is resolved correctly

Next Steps

• Evaluation Results - Understanding output formats
• Architecture - How mcpbr works internally
• Troubleshooting - Common issues and solutions