Best Practices Guide

Best Practices Guide

This guide helps you get the most value from mcpbr while avoiding common pitfalls. Whether you're testing a new MCP server, optimizing costs, or setting up CI/CD pipelines, these practices will help you work effectively.

Quick Reference

Benchmark Selection Guidelines

Choosing the Right Benchmark

Use SWE-bench when:

• Testing code exploration and bug-fixing capabilities
• Evaluating Python-focused MCP servers
• Need proven, standardized benchmarks
• Want fast evaluation with pre-built images

Use CyberGym when:

• Testing security analysis capabilities
• Evaluating C/C++ code understanding
• Need vulnerability detection benchmarks
• Want to test different difficulty levels

SWE-bench Best Practices

Start Small, Scale Gradually

# Step 1: Single task smoke test
mcpbr run -c config.yaml -n 1 -v

# Step 2: Small sample
mcpbr run -c config.yaml -n 5 -o results-5.json

# Step 3: Medium sample
mcpbr run -c config.yaml -n 25 -o results-25.json

# Step 4: Full evaluation (if needed)
mcpbr run -c config.yaml -o results-full.json

Use Pre-built Images

use_prebuilt_images: true  # Much faster and more reliable

Pre-built images provide:

• Validated dependency installation
• Consistent evaluation environment
• Faster startup (no package installation)
• Working Python imports inside containers

Anti-pattern: Disabling pre-built images without good reason

# Avoid this unless debugging dependency issues
mcpbr run -c config.yaml --no-prebuilt

CyberGym Best Practices

Choose Appropriate Difficulty Level

Start with Level 2 for Most Use Cases

# Level 2 provides good balance of challenge and success
mcpbr run -c config.yaml --benchmark cybergym --level 2 -n 5

Increase Timeouts for Compilation

benchmark: cybergym
timeout_seconds: 600  # CyberGym needs more time for builds
max_iterations: 15

Anti-pattern: Using level 3 for all testing

# Level 3 is too easy for meaningful evaluation
mcpbr run -c config.yaml --benchmark cybergym --level 3  # Not recommended

MCP Server Configuration Best Practices

Selecting MCP Servers

Match Server Capabilities to Benchmark Needs

For SWE-bench (bug fixing):

• Filesystem access (read/write)
• Code search capabilities
• Test execution tools
• Git operations

For CyberGym (security):

• Code analysis tools
• AST parsing
• Vulnerability pattern detection
• Build system integration

Configuration Patterns

Good: Clear, Minimal Configuration

mcp_server:
  name: "mcpbr"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]
  env: {}

Better: With Environment Variables

mcp_server:
  name: "codebase"
  command: "npx"
  args: ["-y", "@supermodeltools/mcp-server"]
  env:
    SUPERMODEL_API_KEY: "${SUPERMODEL_API_KEY}"
    LOG_LEVEL: "info"

Anti-pattern: Hardcoded Secrets

mcp_server:
  env:
    API_KEY: "sk-1234..."  # NEVER hardcode secrets!

Testing Your MCP Server

Step 1: Standalone Verification

# Test server starts correctly
npx -y @modelcontextprotocol/server-filesystem /tmp/test

# For custom servers
python -m my_mcp_server --workspace /tmp/test

Step 2: Quick Smoke Test

# Single task with MCP only
mcpbr run -c config.yaml -n 1 -v -M

Step 3: Analyze Tool Usage

# Check if MCP tools are being used
mcpbr run -c config.yaml -n 5 -o results.json
cat results.json | jq '.tasks[0].mcp.tool_usage'

Step 4: Compare Against Baseline

# Full comparison
mcpbr run -c config.yaml -n 10 -o results.json

Red Flags:

• MCP tools never appear in tool_usage
• Tool usage is always 0 or very low
• Similar results between MCP and baseline
• Server startup warnings in logs

Performance Optimization Tips

Docker Resource Management

Set Appropriate Concurrency

# For most systems
max_concurrent: 4

# For powerful machines (16+ GB RAM, 8+ cores)
max_concurrent: 8

# For limited resources or API rate limits
max_concurrent: 2

# For debugging
max_concurrent: 1

Monitor Resource Usage

# Check Docker resource consumption
docker stats

# Check running containers
docker ps | grep mcpbr

Clean Up Orphaned Containers

# Preview what will be removed
mcpbr cleanup --dry-run

# Remove orphaned containers
mcpbr cleanup -f

Apple Silicon Optimization

Expected Performance:

• Tasks take 2-3x longer than native x86_64
• This is normal due to emulation
• Pre-built images help reduce overhead

Recommended Settings:

max_concurrent: 2        # Reduce concurrency
timeout_seconds: 600     # Increase timeouts
use_prebuilt_images: true  # Essential for performance

Install Rosetta 2 (if not already installed)

softwareupdate --install-rosetta

Timeout Tuning

Default Timeouts by Benchmark

Adjust Based on Task Complexity

# Simple tasks
timeout_seconds: 300
max_iterations: 10

# Complex tasks or slow hardware
timeout_seconds: 600
max_iterations: 30

Anti-pattern: Extremely Long Timeouts

timeout_seconds: 3600  # 1 hour - probably too long
max_iterations: 100    # Too many iterations

Model Selection

Development/Testing:

model: "haiku"  # Fast, cheap, good for iteration

Production/Benchmarking:

model: "sonnet"  # Best balance of performance and cost

Maximum Performance:

model: "opus"  # Most capable, highest cost

Cost Management Strategies

Understanding Costs

Token Usage Factors:

• Model choice (Haiku < Sonnet < Opus)
• Number of iterations (more turns = more tokens)
• Task complexity (complex bugs require more exploration)
• Sample size (most obvious cost driver)

Typical Costs (per task, Sonnet model):

• Simple task: $0.10-0.30 (5-10K output tokens)
• Medium task: $0.30-0.80 (10-20K output tokens)
• Complex task: $0.80-2.00 (20-50K output tokens)

Cost Optimization Strategies

1. Start Small

# Test with 1 task first
mcpbr run -c config.yaml -n 1

# Scale to 5 tasks to validate
mcpbr run -c config.yaml -n 5

# Only run full evaluation when confident
mcpbr run -c config.yaml -n 50

2. Use Faster Models for Development

# Development config
model: "haiku"
sample_size: 5
max_iterations: 5
timeout_seconds: 180

3. Skip Baseline During Iteration

# Only run MCP agent while developing
mcpbr run -c config.yaml -M -n 5

4. Reduce Iterations

max_iterations: 10  # Instead of 30

5. Monitor Token Usage

# Save results and check token consumption
mcpbr run -c config.yaml -n 5 -o results.json

# Analyze token usage
cat results.json | jq '.tasks[] | {id: .instance_id, tokens: .mcp.tokens}'

Anti-pattern: Running Full Evaluations Repeatedly

# Bad: Running 300 tasks multiple times during development
mcpbr run -c config.yaml  # Default: full dataset
mcpbr run -c config.yaml  # Oops, again...
# Result: Hundreds of dollars in API costs

Good Pattern: Incremental Testing

# Development cycle
mcpbr run -c config.yaml -n 1 -M  # $0.20
mcpbr run -c config.yaml -n 5 -M  # $1-2
mcpbr run -c config.yaml -n 10    # $5-10
# Only when ready:
mcpbr run -c config.yaml -n 50 -o final.json  # $50-100

Cost Tracking

Track Costs Per Run

import json

with open("results.json") as f:
    results = json.load(f)

# Calculate approximate costs (Sonnet pricing as of 2026)
INPUT_COST = 3.00 / 1_000_000   # $3 per 1M tokens
OUTPUT_COST = 15.00 / 1_000_000  # $15 per 1M tokens

total_cost = 0
for task in results["tasks"]:
    mcp = task.get("mcp", {})
    tokens = mcp.get("tokens", {})
    input_tokens = tokens.get("input", 0)
    output_tokens = tokens.get("output", 0)

    task_cost = (input_tokens * INPUT_COST) + (output_tokens * OUTPUT_COST)
    total_cost += task_cost

print(f"Total cost: ${total_cost:.2f}")
print(f"Average per task: ${total_cost / len(results['tasks']):.2f}")

Result Interpretation Guidelines

Understanding Resolution Rates

What "Resolved" Means:

1. Patch was generated
2. Patch applied cleanly
3. All FAIL_TO_PASS tests now pass
4. All PASS_TO_PASS tests still pass

Interpreting Improvement:

MCP: 32% resolved (8/25)
Baseline: 20% resolved (5/25)
Improvement: +60%

This means:

• MCP agent is 60% better than baseline
• Your MCP server helped on 3 additional tasks
• Both agents struggled (absolute rates are low)

Success Rate Benchmarks

Typical Resolution Rates (SWE-bench Lite):

Low Rates (Both < 15%):

• Tasks may be inherently difficult
• Sample may include hard tasks
• Timeouts may be too short
• Model may need more iterations

High Baseline (> 25%):

• Sample may include easier tasks
• Good task selection
• Model is performing well

Low Improvement (< 10%):

• MCP tools not providing value
• Tools not being used effectively
• Baseline already sufficient

Analyzing Tool Usage

Extract Tool Statistics

cat results.json | jq '.tasks[0].mcp.tool_usage'

Healthy Tool Distribution:

{
  "Grep": 15,        // Searching code
  "Read": 20,        // Reading files
  "Bash": 25,        // Running tests
  "Edit": 5,         // Making changes
  "mcp__read": 10    // MCP tools being used
}

Red Flags:

{
  "TodoWrite": 50,   // Too much planning, not enough action
  "mcp__search": 0   // MCP tools not being used at all
}

Comparing Configurations

Save Results with Descriptive Names

mcpbr run -c filesystem.yaml -o results-filesystem.json
mcpbr run -c supermodel.yaml -o results-supermodel.json

Compare Resolution Rates

import json

def compare_results(file1, file2):
    with open(file1) as f1, open(file2) as f2:
        r1 = json.load(f1)
        r2 = json.load(f2)

    rate1 = r1["summary"]["mcp"]["rate"]
    rate2 = r2["summary"]["mcp"]["rate"]

    print(f"{file1}: {rate1:.1%}")
    print(f"{file2}: {rate2:.1%}")
    print(f"Difference: {(rate2 - rate1):.1%}")

compare_results("results-filesystem.json", "results-supermodel.json")

Security Considerations

API Key Management

Good: Environment Variables

export ANTHROPIC_API_KEY="sk-ant-..."
mcpbr run -c config.yaml

Better: Shell Profile

# Add to ~/.bashrc or ~/.zshrc
export ANTHROPIC_API_KEY="sk-ant-..."

Best: Secret Management

# Using 1Password CLI
export ANTHROPIC_API_KEY=$(op read "op://vault/mcpbr/api_key")

# Using AWS Secrets Manager
export ANTHROPIC_API_KEY=$(aws secretsmanager get-secret-value \
  --secret-id mcpbr/anthropic-key --query SecretString --output text)

Anti-pattern: Hardcoded in Config

# NEVER do this
env:
  ANTHROPIC_API_KEY: "sk-ant-..."  # Will be committed to git!

Docker Security

Network Isolation (when external access not needed)

# For most use cases, network access is required for API calls
# But if testing without MCP:
docker_network_mode: "none"

Container Cleanup

# Regularly clean up orphaned containers
mcpbr cleanup -f

# Check for running containers
docker ps | grep mcpbr

Data Security

Sensitive Repositories:

• mcpbr runs on public datasets (SWE-bench, CyberGym)
• Do NOT use with proprietary code
• Task data is sent to Anthropic API
• Logs may contain code snippets

Log Management:

# Logs contain code and conversations
mcpbr run -c config.yaml --log-dir logs/

# Secure log files
chmod 700 logs/

CI/CD Integration Patterns

GitHub Actions

Basic Workflow

name: MCP Benchmark

on:
  pull_request:
    paths:
      - 'mcp-server/**'

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install mcpbr
        run: pip install mcpbr

      - name: Run benchmark
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          mcpbr run -c config.yaml -n 10 -o results.json

      - name: Upload results
        uses: actions/upload-artifact@v3
        with:
          name: benchmark-results
          path: results.json

With Regression Detection

- name: Download baseline
  run: |
    gh run download --name baseline-results --dir .

- name: Run with regression detection
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    mcpbr run -c config.yaml -n 25 \
      --baseline-results baseline.json \
      --regression-threshold 0.1 \
      --slack-webhook ${{ secrets.SLACK_WEBHOOK }} \
      -o current.json

With JUnit XML

- name: Run benchmark
  run: mcpbr run -c config.yaml --output-junit junit.xml

- name: Publish test results
  uses: EnricoMi/publish-unit-test-result-action@v2
  if: always()
  with:
    files: junit.xml

GitLab CI

mcpbr-benchmark:
  image: python:3.11
  services:
    - docker:dind
  variables:
    DOCKER_HOST: tcp://docker:2375
  script:
    - pip install mcpbr
    - mcpbr run -c config.yaml -n 10 --output-junit junit.xml
  artifacts:
    reports:
      junit: junit.xml
    paths:
      - results.json
  only:
    - merge_requests

Cost Control in CI/CD

Sample Size Limits

# Don't run full benchmarks on every PR
mcpbr run -c config.yaml -n 10  # Small sample for PRs

# Full benchmarks on main branch only
mcpbr run -c config.yaml -n 50  # Larger sample for releases

Conditional Execution

# Only run on MCP server changes
on:
  push:
    paths:
      - 'mcp-server/**'
      - 'config.yaml'

Debugging and Troubleshooting

Diagnostic Workflow

Step 1: Verify Prerequisites

# Check Docker
docker info

# Check API key
echo $ANTHROPIC_API_KEY | head -c 10

# Check Claude CLI
which claude

Step 2: Test MCP Server Standalone

# For filesystem server
npx -y @modelcontextprotocol/server-filesystem /tmp/test

# For custom servers
python -m my_mcp_server --workspace /tmp/test

Step 3: Run Single Task with Verbose Logging

mcpbr run -c config.yaml -n 1 -vv --log-dir debug/

Step 4: Analyze Logs

# Check system events
cat debug/*.json | jq '.events[] | select(.type == "system")'

# Check tool usage
cat debug/*.json | jq '.events[] | select(.type == "assistant") |
  .message.content[] | select(.type == "tool_use") | .name'

Common Issues and Solutions

MCP Server Not Starting

Warning: MCP server add failed (exit 1)

Solutions:

1. Test server command directly
2. Check environment variables are set
3. Verify command is in PATH
4. Check server logs for errors

No Patch Generated

No changes made by Claude Code

Causes:

• Task too complex for iterations limit
• Agent couldn't find solution
• Agent made changes then reverted

Solutions:

max_iterations: 30  # Increase from 10
timeout_seconds: 600  # Increase from 300

Timeouts

Timeout after 300 seconds

Solutions:

timeout_seconds: 600
max_concurrent: 2  # Reduce load

Tests Failing

FAIL_TO_PASS: 0/2 passed

This means:

• Patch applied successfully
• But didn't fix the bug
• Agent made incorrect changes
• Not an mcpbr issue - agent behavior

Debug Flags

Verbose Output Levels

# Standard output
mcpbr run -c config.yaml

# Verbose: summary + task progress
mcpbr run -c config.yaml -v

# Very verbose: detailed tool calls
mcpbr run -c config.yaml -vv

Per-Instance Logs

# Create detailed JSON logs for each task
mcpbr run -c config.yaml --log-dir logs/

# Logs are timestamped: instance_id_runtype_timestamp.json
ls logs/
# django__django-11099_mcp_20260120_143052.json
# django__django-11099_baseline_20260120_143156.json

Single Log File

# All events in one file
mcpbr run -c config.yaml --log-file full.log

Iterative Development Workflow

Phase 1: Quick Validation

Goal: Verify basic functionality

# 1. Test MCP server starts
npx -y @modelcontextprotocol/server-filesystem /tmp/test

# 2. Run single task
mcpbr init -t quick-test
mcpbr run -c mcpbr.yaml -v

# 3. Check if it worked
# - Did the server start? (check for warnings)
# - Were tools registered? (check verbose output)
# - Was a patch generated?

Success Criteria:

• No server startup errors
• Task completes without timeout
• Patch generated (even if incorrect)

Phase 2: Small-Scale Testing

Goal: Validate at small scale

# 1. Run 5 tasks with MCP only
mcpbr run -c config.yaml -n 5 -M -o dev-mcp.json

# 2. Analyze tool usage
cat dev-mcp.json | jq '.tasks[].mcp.tool_usage'

# 3. Check if MCP tools are used
cat dev-mcp.json | jq '.tasks[].mcp.tool_usage |
  to_entries | map(select(.key | startswith("mcp")))'

Success Criteria:

• MCP tools appear in tool_usage
• At least 1-2 tasks resolved
• No consistent errors

Phase 3: Baseline Comparison

Goal: Measure improvement

# 1. Run 10 tasks with MCP + baseline
mcpbr run -c config.yaml -n 10 -o comparison.json

# 2. Check improvement
cat comparison.json | jq '.summary'

# 3. Find MCP-only wins
cat comparison.json | jq '.tasks[] |
  select(.mcp.resolved == true and .baseline.resolved == false) |
  .instance_id'

Success Criteria:

• MCP rate > baseline rate
• At least 1-2 MCP-only wins
• Improvement > 10%

Phase 4: Optimization

Goal: Improve performance based on findings

Analyze Failures:

# Find tasks where MCP failed
cat comparison.json | jq '.tasks[] |
  select(.mcp.resolved == false) |
  {id: .instance_id, error: .mcp.error, iterations: .mcp.iterations}'

Common Optimizations:

• Increase iterations if hitting limits
• Adjust timeout if tasks timeout
• Modify MCP server configuration
• Update agent prompt

Phase 5: Production Evaluation

Goal: Final comprehensive benchmark

# 1. Run larger sample
mcpbr run -c config.yaml -n 50 -o production.json -r report.md

# 2. Save for regression detection
cp production.json baseline.json

# 3. Generate all outputs
mcpbr run -c config.yaml -n 50 \
  -o results.json \
  -y results.yaml \
  -r report.md \
  --output-junit junit.xml \
  --log-dir logs/

Success Criteria:

• Statistically significant sample (n >= 25)
• Results saved for future comparison
• Improvement is consistent
• Documentation completed

Templates and Configuration

Using Templates Effectively

Start with Templates

# List all templates
mcpbr templates

# Use appropriate template
mcpbr init -t quick-test      # For testing
mcpbr init -t filesystem      # For development
mcpbr init -t production      # For final evaluation

Customize After Generation

# Generate from template
mcpbr init -t filesystem

# Edit to customize
vim mcpbr.yaml

# Test your changes
mcpbr run -c mcpbr.yaml -n 1 -v

Configuration Patterns

Development Configuration

mcp_server:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]

model: "haiku"              # Fast and cheap
sample_size: 5              # Small sample
max_concurrent: 1           # Serial execution for debugging
timeout_seconds: 180        # Shorter timeout
max_iterations: 5           # Fewer iterations
use_prebuilt_images: true

Production Configuration

mcp_server:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]

model: "sonnet"             # Best balance
sample_size: 50             # Meaningful sample
max_concurrent: 4           # Parallel execution
timeout_seconds: 600        # Generous timeout
max_iterations: 30          # More iterations
use_prebuilt_images: true

CI/CD Configuration

mcp_server:
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]

model: "sonnet"
sample_size: 10             # Quick feedback
max_concurrent: 2           # Don't overload
timeout_seconds: 300
max_iterations: 15          # Balanced
use_prebuilt_images: true

Examples and Use Cases

Use Case 1: Testing a New MCP Server

Scenario: You've built a custom MCP server with advanced code search

Workflow:

# 1. Create config from template
mcpbr init -t custom-python -o my-server.yaml

# 2. Edit to point to your server
# Update args: ["-m", "my_mcp_server", "--workspace", "{workdir}"]

# 3. Test standalone
python -m my_mcp_server --workspace /tmp/test

# 4. Quick test
mcpbr run -c my-server.yaml -n 1 -v -M

# 5. Small comparison
mcpbr run -c my-server.yaml -n 10 -o results.json

# 6. Analyze tool usage
cat results.json | jq '.tasks[0].mcp.tool_usage'

# 7. Full evaluation if promising
mcpbr run -c my-server.yaml -n 50 -o final.json -r report.md

Use Case 2: Comparing Two MCP Servers

Scenario: Evaluating filesystem vs. Supermodel

Workflow:

# 1. Create configurations
mcpbr init -t filesystem -o filesystem.yaml
mcpbr init -t supermodel -o supermodel.yaml

# 2. Set API key for Supermodel
export SUPERMODEL_API_KEY="your-key"

# 3. Run identical samples
mcpbr run -c filesystem.yaml -n 25 -o fs-results.json
mcpbr run -c supermodel.yaml -n 25 -o sm-results.json

# 4. Compare results
python compare.py fs-results.json sm-results.json

# 5. Analyze differences
# Check which tasks each solved
# Compare tool usage patterns
# Analyze token consumption

Use Case 3: Cost-Optimized Development

Scenario: Limited budget, need to test iteratively

Workflow:

# Phase 1: Ultra-cheap validation (< $1)
mcpbr init -t quick-test
# Edit: model: "haiku", sample_size: 1, max_iterations: 3
mcpbr run -c mcpbr.yaml -M  # MCP only, ~$0.10

# Phase 2: Small test (< $5)
# Edit: sample_size: 5, max_iterations: 5
mcpbr run -c mcpbr.yaml -M  # ~$2-3

# Phase 3: Baseline comparison (< $20)
# Edit: sample_size: 10, max_iterations: 10
mcpbr run -c mcpbr.yaml -o results.json  # ~$10-15

# Phase 4: Production (budgeted)
# Switch to: model: "sonnet", sample_size: 50
mcpbr run -c mcpbr.yaml -o final.json  # ~$100-150

Use Case 4: CI/CD Integration

Scenario: Automated regression testing on PR

Workflow:

# .github/workflows/mcp-test.yml
name: MCP Regression Test

on:
  pull_request:
    paths: ['mcp-server/**']

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Download baseline
        run: gh run download --name baseline --dir .

      - name: Install mcpbr
        run: pip install mcpbr

      - name: Run regression test
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          mcpbr run -c config.yaml -n 10 \
            --baseline-results baseline.json \
            --regression-threshold 0.1 \
            --slack-webhook ${{ secrets.SLACK_WEBHOOK }} \
            -o current.json \
            --output-junit junit.xml

      - name: Publish results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: always()
        with:
          files: junit.xml

Use Case 5: Security Research

Scenario: Evaluating vulnerability detection capabilities

Workflow:

# 1. Start with basic level
mcpbr init -t cybergym-basic

# 2. Test single vulnerability
mcpbr run -c mcpbr.yaml -n 1 -v --log-dir logs/

# 3. Check PoC generation
# Look for poc.c, poc.py in logs
cat logs/*.json | jq '.events[] | select(.type == "assistant") |
  .message.content[] | select(.type == "text") | .text' | grep -i poc

# 4. Scale up
mcpbr run -c mcpbr.yaml -n 5 -o level1.json

# 5. Try higher level
mcpbr init -t cybergym-advanced -o level3.yaml
mcpbr run -c level3.yaml -n 5 -o level3.json

# 6. Compare difficulty levels
python compare.py level1.json level3.json

Anti-Patterns to Avoid

Configuration Anti-Patterns

Bad: Hardcoded secrets

mcp_server:
  env:
    API_KEY: "sk-1234..."  # Will be committed!

Bad: Unrealistic timeouts

timeout_seconds: 3600  # 1 hour
max_iterations: 100

Bad: Excessive concurrency

max_concurrent: 20  # Will hit rate limits

Workflow Anti-Patterns

Bad: Running full benchmark during development

# Don't do this repeatedly during development
mcpbr run -c config.yaml  # Full dataset every time!

Bad: Not saving results

# No way to compare or track progress
mcpbr run -c config.yaml  # Results lost!

Bad: Skipping standalone testing

# MCP server fails, wasting API costs
mcpbr run -c broken-config.yaml -n 50

Analysis Anti-Patterns

Bad: Focusing only on resolution rate

# Missing important insights
rate = results["summary"]["mcp"]["rate"]
print(f"Rate: {rate}")  # That's it?

Bad: Not checking tool usage

# MCP tools might not be used at all!
print(f"Resolved: {results['summary']['mcp']['resolved']}")

Bad: Comparing different samples

# Results not comparable
mcpbr run -c config-a.yaml -n 10  # Random 10
mcpbr run -c config-b.yaml -n 10  # Different random 10

Quick Start Checklist

Before First Run:

• Docker installed and running
• ANTHROPIC_API_KEY set
• Claude CLI installed (which claude)
• mcpbr installed (mcpbr --version)

For New MCP Server:

• Test server standalone
• Create config from template
• Run single task test (n=1)
• Check tool registration
• Verify MCP tools used
• Scale to 5-10 tasks
• Save results
• Compare vs baseline

For Production Run:

• Config validated
• Sample size determined
• Timeout appropriate
• Output paths specified
• Baseline results saved
• Budget confirmed
• Results will be saved

Security Best Practices

Securing your mcpbr deployment is critical, especially when running evaluations in CI/CD pipelines, shared environments, or with third-party MCP servers.

API Key Management

!!! danger "Never commit API keys to version control" API keys in configuration files or source code are one of the most common security incidents. Use environment variables or secret management tools exclusively.

Hierarchy of Key Management (from basic to advanced):

Using .env files with mcpbr:

mcpbr automatically loads .env files from the current directory. Create one but ensure it is never committed:

# Create .env file
echo 'ANTHROPIC_API_KEY=sk-ant-...' > .env
echo 'SUPERMODEL_API_KEY=your-key-here' >> .env

# Ensure .env is gitignored
echo '.env' >> .gitignore

Then reference variables in your config:

mcp_server:
  env:
    SUPERMODEL_API_KEY: "${SUPERMODEL_API_KEY}"

Using secret managers in CI/CD:

# AWS Secrets Manager
export ANTHROPIC_API_KEY=$(aws secretsmanager get-secret-value \
  --secret-id mcpbr/anthropic-key --query SecretString --output text)

# 1Password CLI
export ANTHROPIC_API_KEY=$(op read "op://vault/mcpbr/api_key")

# HashiCorp Vault
export ANTHROPIC_API_KEY=$(vault kv get -field=api_key secret/mcpbr)

!!! tip "Rotate keys regularly" Set a reminder to rotate your API keys on a regular schedule (e.g., every 90 days). If you suspect a key has been exposed, rotate it immediately in the Anthropic Console.

Docker Security

mcpbr runs evaluation tasks inside Docker containers, which provides a baseline of isolation. You can harden this further.

Resource limits to prevent runaway containers:

# In your config, control concurrency to bound resource usage
max_concurrent: 4          # Limit parallel containers
timeout_seconds: 600       # Hard timeout per task
max_iterations: 30         # Cap agent turns

Monitor container activity:

# Watch container resource usage in real time
docker stats --filter "name=mcpbr"

# List all mcpbr containers (including stopped)
docker ps -a --filter "name=mcpbr"

Network isolation for non-API workloads:

# If your MCP server does not need network access, isolate it
docker_network_mode: "none"

!!! warning "Containers run as root by default" Docker containers in mcpbr run as root within the container to ensure dependency installation and test execution work correctly. This is isolated from the host via Docker's namespacing, but avoid mounting sensitive host directories as volumes.

Volume mount security:

# Only mount what is necessary
volumes:
  "/path/to/cache": "/cache"  # Read-write mount for caching

# NEVER mount these:
# volumes:
#   "/etc": "/host-etc"        # Host system configuration
#   "/var/run/docker.sock": "/var/run/docker.sock"  # Docker socket
#   "~/.ssh": "/root/.ssh"     # SSH keys

MCP Server Sandboxing

!!! warning "Third-party MCP servers execute arbitrary code" MCP servers run commands and access files within the Docker container. Only use MCP servers you trust, and review their source code before deploying in production.

Sandboxing recommendations:

1. Review server source code before first use
2. Pin server versions to prevent supply-chain attacks:

   mcp_server:
     command: "npx"
     args: ["-y", "@modelcontextprotocol/server-filesystem@1.2.3", "{workdir}"]
   

3. Limit environment variables exposed to the server -- only pass what is required
4. Use setup_command cautiously -- it runs with full container privileges

Output Sanitization

Logs and results may contain sensitive data:

• Task logs can include code snippets from repositories
• API conversations are recorded in per-instance logs
• Results JSON contains tool call traces

Secure your outputs:

# Restrict log directory permissions
mcpbr run -c config.yaml --log-dir logs/
chmod 700 logs/

# Redact sensitive fields before sharing results
cat results.json | jq 'del(.tasks[].mcp.conversation)' > results-safe.json

!!! tip "Audit before sharing" Before sharing results files, reports, or logs externally, review them for any API keys, internal paths, or proprietary code that may have been captured during evaluation.

Secure CI/CD Pipeline Configuration

# .github/workflows/benchmark.yml
name: MCP Benchmark (Secure)

on:
  pull_request:
    paths: ['mcp-server/**']

permissions:
  contents: read  # Minimal permissions

jobs:
  benchmark:
    runs-on: ubuntu-latest
    environment: benchmarks  # Use a protected environment
    steps:
      - uses: actions/checkout@v4

      - name: Install mcpbr
        run: pip install mcpbr

      - name: Run benchmark
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          mcpbr run -c config.yaml -n 10 -o results.json

      - name: Upload results (restricted retention)
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-results
          path: results.json
          retention-days: 30  # Don't keep results forever

CI/CD security checklist:

• API keys stored as repository or environment secrets (never in workflow files)
• Workflow permissions set to minimum required (contents: read)
• Protected environments for production benchmarks
• Artifact retention policies configured
• Branch protection rules on main branch
• Audit logs enabled for secret access

Performance Optimization

Beyond the basic performance tips covered earlier, this section provides advanced optimization strategies for large-scale evaluations.

Concurrent Task Execution

The max_concurrent setting controls how many Docker containers run simultaneously. Tuning this requires balancing CPU, memory, network bandwidth, and API rate limits.

Recommended settings by machine profile:

# Conservative (safe default)
max_concurrent: 4

# Aggressive (high-end hardware with sufficient API quota)
max_concurrent: 8

!!! tip "Monitor and adjust" Start with max_concurrent: 4 and monitor with docker stats. If you see containers being OOM-killed or Docker becoming unresponsive, reduce concurrency. If CPU and memory utilization are low, increase it.

Docker Image Caching and Pre-built Images

Docker image pulls and builds are often the largest time cost for first-time runs. Optimize this with caching strategies.

Always use pre-built images:

use_prebuilt_images: true  # Default and recommended

Pre-pull images before evaluation:

# Pre-pull common base images to avoid per-task download delays
docker pull ghcr.io/epoch-research/swe-bench.eval.x86_64.django__django-11099
docker pull ghcr.io/epoch-research/swe-bench.eval.x86_64.astropy__astropy-12907

Docker build cache optimization:

# Ensure Docker build cache is not pruned aggressively
docker system df  # Check disk usage

# Prune only dangling images, keep build cache
docker image prune -f  # Remove dangling only

!!! warning "Disk space management" SWE-bench pre-built images are large (1-3 GB each). A full evaluation may require 50+ GB of Docker image storage. Monitor disk usage with docker system df and clean up between runs with mcpbr cleanup.

Dataset Caching

mcpbr supports result caching to avoid re-running identical evaluations. This is especially valuable during iterative development.

# Enable caching
cache_enabled: true
cache_dir: "/path/to/cache"  # Default: ~/.cache/mcpbr

When caching helps most:

• Re-running after a configuration change that only affects one side (MCP or baseline)
• Iterating on MCP server changes with the same task set
• Resuming after a partial failure

Persistent volume caching for MCP servers:

If your MCP server performs expensive pre-computation (like codebase indexing), use the setup_command and volumes features:

mcp_server:
  command: "npx"
  args: ["-y", "@supermodeltools/mcp-server", "{workdir}"]
  setup_command: "npx -y @supermodeltools/mcp-server --index {workdir}"
  setup_timeout_ms: 900000  # 15 minutes for indexing

# Mount a persistent volume for caching across tasks
volumes:
  "/tmp/mcpbr-cache": "/cache"

Memory Management for Large Benchmarks

Full dataset evaluations (300+ tasks) can strain system memory. Use these strategies to stay within bounds.

# Reduce concurrency to lower peak memory
max_concurrent: 2

# Use the graceful degradation system to handle failures
continue_on_error: true
max_failures: 10  # Stop if too many tasks fail (likely a systemic issue)

# Enable checkpointing for crash recovery
checkpoint_interval: 1  # Save state after every task

System-level memory monitoring:

# Monitor Docker memory usage
docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"

# Set Docker Desktop memory limit (macOS/Windows)
# Docker Desktop > Settings > Resources > Memory: 8-12 GB

!!! example "Memory budget rule of thumb" Each concurrent container uses approximately 1-3 GB of RAM. For a machine with 16 GB total, allocate 8-12 GB to Docker and set max_concurrent to 3-4.

Network Optimization for Remote Evaluations

Reduce network latency:

• Use a cloud VM in the same region as the Anthropic API endpoint
• Pre-pull Docker images before starting evaluations
• Use setup_command to front-load network-intensive operations

Azure infrastructure mode for large runs:

infrastructure:
  mode: azure
  azure:
    resource_group: "mcpbr-eval"
    location: "eastus"  # Close to Anthropic API
    cpu_cores: 8
    memory_gb: 32
    disk_gb: 250
    auto_shutdown: true

CI/CD Integration

This section provides production-ready CI/CD configurations for automated benchmarking.

GitHub Actions Example Workflow

Complete workflow with caching, regression detection, and notifications:

name: MCP Server Benchmark

on:
  pull_request:
    paths: ['mcp-server/**', 'config.yaml']
  schedule:
    - cron: '0 6 * * 1'  # Weekly on Monday at 6 AM UTC

concurrency:
  group: benchmark-${{ github.ref }}
  cancel-in-progress: true

jobs:
  benchmark:
    runs-on: ubuntu-latest
    timeout-minutes: 120
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python 3.11
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install mcpbr
        run: pip install mcpbr

      - name: Cache Docker images
        uses: actions/cache@v4
        with:
          path: /tmp/docker-cache
          key: docker-${{ runner.os }}-${{ hashFiles('config.yaml') }}
          restore-keys: docker-${{ runner.os }}-

      - name: Load cached Docker images
        run: |
          if [ -d /tmp/docker-cache ]; then
            for img in /tmp/docker-cache/*.tar; do
              docker load -i "$img" 2>/dev/null || true
            done
          fi

      - name: Download baseline results
        uses: actions/download-artifact@v4
        with:
          name: baseline-results
          path: .
        continue-on-error: true  # OK if no baseline exists yet

      - name: Run benchmark
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          ARGS="-c config.yaml -n 10 -o results.json --output-junit junit.xml"
          if [ -f baseline.json ]; then
            ARGS="$ARGS --baseline-results baseline.json --regression-threshold 0.1"
          fi
          mcpbr run $ARGS

      - name: Publish test results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: always()
        with:
          files: jun

…(truncated)