1910 Comparison Mode Ab48989e

Side-by-Side MCP Server Comparison

Compare two MCP servers head-to-head in a single evaluation run to understand which implementation performs better on your benchmark tasks.

Overview

Comparison mode enables A/B testing of MCP servers by running the same tasks against two different server implementations simultaneously. This is useful for:

• Evaluating implementation alternatives: Compare different MCP server architectures
• Testing optimizations: Measure the impact of performance improvements
• Benchmarking tools: Compare your MCP server against reference implementations
• Research: Conduct controlled experiments on MCP server design decisions

Quick Start

1. Create a Comparison Configuration

Create a YAML config file (e.g., comparison.yaml):

comparison_mode: true

mcp_server_a:
  name: "Task Queries"
  command: node
  args: [build/index.js]
  cwd: /path/to/task-queries-server

mcp_server_b:
  name: "Edge Identity"
  command: node
  args: [build/index.js]
  cwd: /path/to/edge-identity-server

benchmark: swe-bench-lite
sample_size: 10
timeout_seconds: 300
max_iterations: 10

2. Run the Comparison

mcpbr run -c comparison.yaml -o results.json

3. View Results

The output will show:

Side-by-Side MCP Server Comparison

┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Metric            ┃ Task Queries ┃ Edge Identity┃ Δ (A - B)┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Resolved Tasks    │ 4/10         │ 2/10         │ +2       │
│ Resolution Rate   │ 40.0%        │ 20.0%        │ +100.0%  │
└───────────────────┴──────────────┴──────────────┴──────────┘

Per-Task Results
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Instance ID ┃ Task Queries ┃ Edge Identity┃ Winner       ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ task-1      │ PASS         │ FAIL         │ Task Queries │
│ task-2      │ PASS         │ FAIL         │ Task Queries │
│ task-3      │ PASS         │ PASS         │ Both         │
└─────────────┴──────────────┴──────────────┴──────────────┘

✓ Task Queries unique wins: 2 tasks
  - task-1
  - task-2

Configuration

Required Fields

When comparison_mode: true, you must provide:

• mcp_server_a: Configuration for the first MCP server
• mcp_server_b: Configuration for the second MCP server

Both servers support the same configuration options as the standard mcp_server field:

mcp_server_a:
  name: "Server Name"           # Human-readable name (appears in reports)
  command: "npx"                 # Command to start the server
  args: ["-y", "package-name"]   # Command arguments
  env:                           # Environment variables (optional)
    API_KEY: "${API_KEY}"
  cwd: "/path/to/server"         # Working directory (optional)
  startup_timeout_ms: 60000      # Server startup timeout (optional)
  tool_timeout_ms: 900000        # Tool call timeout (optional)

Complete Example

# Enable comparison mode
comparison_mode: true

# First MCP server
mcp_server_a:
  name: "Optimized Server"
  command: node
  args: [dist/server.js]
  cwd: /Users/me/projects/optimized-mcp
  env:
    LOG_LEVEL: debug
    CACHE_ENABLED: "true"

# Second MCP server
mcp_server_b:
  name: "Baseline Server"
  command: node
  args: [dist/server.js]
  cwd: /Users/me/projects/baseline-mcp
  env:
    LOG_LEVEL: debug
    CACHE_ENABLED: "false"

# Model and provider (shared by both servers)
provider: anthropic
model: claude-sonnet-4-20250514

# Benchmark settings
benchmark: swe-bench-verified
sample_size: 50
timeout_seconds: 600
max_iterations: 15

Output Format

JSON Results

The results JSON includes comparison-specific fields:

{
  "metadata": {
    "timestamp": "2026-01-30T10:00:00Z",
    "config": {
      "comparison_mode": true,
      ...
    },
    "mcp_server_a": {
      "name": "Task Queries",
      "command": "node",
      "args": ["build/index.js"]
    },
    "mcp_server_b": {
      "name": "Edge Identity",
      "command": "node",
      "args": ["build/index.js"]
    }
  },
  "summary": {
    "mcp_server_a": {
      "name": "Task Queries",
      "resolved": 4,
      "total": 10,
      "resolution_rate": 0.4,
      "cost": 0.50,
      "tool_calls": 125
    },
    "mcp_server_b": {
      "name": "Edge Identity",
      "resolved": 2,
      "total": 10,
      "resolution_rate": 0.2,
      "cost": 0.45,
      "tool_calls": 98
    },
    "comparison": {
      "a_vs_b_delta": 2,
      "a_vs_b_improvement_pct": 100.0,
      "a_unique_wins": ["task-1", "task-2"],
      "b_unique_wins": [],
      "both_wins": ["task-3"],
      "both_fail": ["task-4", "task-5", ...]
    }
  },
  "tasks": [
    {
      "instance_id": "task-1",
      "mcp_server_a": {
        "resolved": true,
        "patch_generated": true,
        "cost": 0.05,
        ...
      },
      "mcp_server_b": {
        "resolved": false,
        "patch_generated": false,
        "cost": 0.04,
        ...
      }
    },
    ...
  ]
}

Understanding the Metrics

• Resolution Rate: Percentage of tasks successfully resolved by each server
• Δ (A - B): Difference in resolved tasks (positive means A is better)
• Improvement %: Percentage improvement of A over B
• Unique Wins: Tasks where only one server succeeded
• Both Wins: Tasks where both servers succeeded
• Both Fail: Tasks where both servers failed

Use Cases

Comparing MCP Tools

Test which tool implementations work best for your use case:

comparison_mode: true

mcp_server_a:
  name: "Filesystem Tools"
  command: npx
  args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]

mcp_server_b:
  name: "Git Tools"
  command: npx
  args: ["-y", "@modelcontextprotocol/server-git", "{workdir}"]

benchmark: swe-bench-lite
sample_size: 20

Performance Optimization

Measure the impact of caching, indexing, or other optimizations:

comparison_mode: true

mcp_server_a:
  name: "With Cache"
  command: node
  args: [server.js]
  env:
    ENABLE_CACHE: "true"

mcp_server_b:
  name: "Without Cache"
  command: node
  args: [server.js]
  env:
    ENABLE_CACHE: "false"

Version Comparison

Compare different versions of the same server:

comparison_mode: true

mcp_server_a:
  name: "v2.0"
  command: node
  args: [server.js]
  cwd: /path/to/v2.0

mcp_server_b:
  name: "v1.5"
  command: node
  args: [server.js]
  cwd: /path/to/v1.5

Limitations

Resource Usage

• 2x Execution Time: Each task runs twice (once per server)
• 2x API Costs: Both servers make independent API calls
• 2x Compute: Both servers run sequentially (no parallel execution)

Budget Tracking

When using budget parameter, costs from both servers count toward the limit:

comparison_mode: true
budget: 10.00  # Shared budget for both servers

Baseline Evaluation

Comparison mode is compatible with baseline evaluation:

# Run both servers + baseline
mcpbr run -c comparison.yaml

# Run only comparison (skip baseline)
mcpbr run -c comparison.yaml -M

Best Practices

1. Use Meaningful Names

Choose descriptive names that clearly identify what's being compared:

mcp_server_a:
  name: "GPT-4 Index"  # Good: specific and clear

mcp_server_b:
  name: "Server B"     # Bad: generic

2. Start Small

Begin with a small sample size to verify your configuration:

comparison_mode: true
sample_size: 5  # Start with 5 tasks

Then scale up once you've validated the setup.

3. Control Variables

Change only one thing at a time for meaningful comparisons:

# Good: Only caching differs
mcp_server_a:
  command: node
  args: [server.js]
  env: {CACHE: "on"}

mcp_server_b:
  command: node
  args: [server.js]
  env: {CACHE: "off"}

# Bad: Multiple differences (harder to interpret)
mcp_server_a:
  command: python
  args: [new_server.py]
  env: {CACHE: "on", LOG_LEVEL: "debug"}

mcp_server_b:
  command: node
  args: [old_server.js]
  env: {CACHE: "off", LOG_LEVEL: "info"}

4. Run Multiple Trials

For statistical confidence, run multiple evaluations:

for i in {1..5}; do
  mcpbr run -c comparison.yaml --trial-mode -o trial_${i}.json
done

Backward Compatibility

Comparison mode is fully backward compatible. Existing single-server configurations continue to work:

# Old config (still works)
mcp_server:
  command: npx
  args: ["-y", "server"]

# New config (comparison mode)
comparison_mode: true
mcp_server_a:
  command: npx
  args: ["-y", "server-a"]
mcp_server_b:
  command: npx
  args: ["-y", "server-b"]

Troubleshooting

"requires both mcp_server_a and mcp_server_b"

You enabled comparison_mode: true but only provided one server. Both servers are required.

"use mcp_server_a/b instead of mcp_server"

You enabled comparison mode but also provided the legacy mcp_server field. Remove it:

comparison_mode: true
# mcp_server: ...  # Remove this
mcp_server_a: ...
mcp_server_b: ...

"mcp_server_a/b only valid in comparison_mode"

You provided comparison servers but didn't enable comparison mode:

comparison_mode: true  # Add this
mcp_server_a: ...
mcp_server_b: ...

Examples

See the examples/comparison/ directory for complete working examples.