SDK Reference

SDK Reference

The mcpbr.sdk module provides the public Python SDK for programmatic access to MCP server benchmarking. It is the primary entry point for Python users who want to configure, validate, and execute benchmarks without the CLI.

All public symbols are re-exported from the top-level mcpbr package.

from mcpbr import MCPBenchmark, BenchmarkResult
from mcpbr import list_benchmarks, list_models, list_providers, get_version

MCPBenchmark

The main class for configuring and running MCP benchmarks.

::: mcpbr.sdk.MCPBenchmark options: show_root_heading: true show_source: false members: - init - validate - dry_run - run

Initialization

MCPBenchmark accepts three types of configuration input:

=== "From a dict"

```python
from mcpbr import MCPBenchmark

bench = MCPBenchmark({
    "mcp_server": {
        "command": "npx",
        "args": ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"],
    },
    "benchmark": "humaneval",
    "model": "sonnet",
    "sample_size": 10,
    "timeout_seconds": 300,
})
```

=== "From a YAML file"

```python
from mcpbr import MCPBenchmark

bench = MCPBenchmark("mcpbr.yaml")
# or with a Path object
from pathlib import Path
bench = MCPBenchmark(Path("configs/production.yaml"))
```

=== "From a HarnessConfig"

```python
from mcpbr import MCPBenchmark
from mcpbr.config import HarnessConfig, MCPServerConfig

config = HarnessConfig(
    mcp_server=MCPServerConfig(
        command="uvx",
        args=["my-mcp-server", "--workdir", "{workdir}"],
    ),
    benchmark="swe-bench-verified",
    model="sonnet",
    sample_size=5,
)
bench = MCPBenchmark(config)
```

!!! warning "File Not Found" When passing a file path, MCPBenchmark raises FileNotFoundError if the file does not exist. When passing a dict, it raises ValueError if the configuration is invalid.

validate()

Check that the configuration is internally consistent before running.

bench = MCPBenchmark({
    "mcp_server": {"command": "npx", "args": ["my-server"]},
    "benchmark": "humaneval",
    "model": "sonnet",
})

is_valid, errors = bench.validate()
if not is_valid:
    for error in errors:
        print(f"Validation error: {error}")
else:
    print("Configuration is valid")

Returns: tuple[bool, list[str]] -- A tuple of (is_valid, list_of_warnings_or_errors).

Validation checks include:

dry_run()

Generate an execution plan without running anything. Useful for previewing what would happen.

plan = bench.dry_run()
print(plan)
# {
#     "benchmark": "humaneval",
#     "model": "sonnet",
#     "provider": "anthropic",
#     "agent_harness": "claude-code",
#     "timeout_seconds": 300,
#     "max_concurrent": 4,
#     "max_iterations": 10,
#     "sample_size": 10,
#     "mcp_server": {
#         "command": "npx",
#         "args": ["my-server"],
#         "name": "mcpbr",
#     },
# }

Returns: dict[str, Any] -- A dictionary describing the execution plan, including benchmark, model, provider, MCP server config, and runtime settings.

The plan includes comparison mode information when comparison_mode is enabled:

bench = MCPBenchmark({
    "comparison_mode": True,
    "mcp_server_a": {"command": "server-a", "name": "Server A"},
    "mcp_server_b": {"command": "server-b", "name": "Server B"},
    "benchmark": "humaneval",
    "model": "sonnet",
})
plan = bench.dry_run()
# plan["comparison_mode"] == True
# plan["mcp_server_a"] and plan["mcp_server_b"] are present

run()

Execute the benchmark asynchronously.

import asyncio
from mcpbr import MCPBenchmark

bench = MCPBenchmark({
    "mcp_server": {"command": "npx", "args": ["my-server", "{workdir}"]},
    "benchmark": "humaneval",
    "model": "sonnet",
})

# Run asynchronously
result = asyncio.run(bench.run())
print(result.success, result.summary)

!!! note "Execution Status" Full benchmark execution via the SDK run() method delegates to an internal _execute() method. Currently, _execute() raises NotImplementedError -- use the mcpbr CLI for actual benchmark runs, or mock MCPBenchmark._execute for testing.

Returns: BenchmarkResult -- A dataclass with the evaluation results.

BenchmarkResult

Dataclass representing the result of a benchmark run.

::: mcpbr.sdk.BenchmarkResult options: show_root_heading: true show_source: false

Fields

!!! example "Working with BenchmarkResult" ```python result = BenchmarkResult( success=True, summary={"pass_rate": 0.85, "resolved": 17, "total": 20}, tasks=[{"task_id": "task_1", "resolved": True}, ...], metadata={"benchmark": "humaneval", "model": "sonnet"}, total_cost=1.23, total_tokens=150000, duration_seconds=245.7, )

if result.success:
    print(f"Pass rate: {result.summary['pass_rate']:.0%}")
    print(f"Cost: ${result.total_cost:.2f}")
    print(f"Duration: {result.duration_seconds:.0f}s")
```

Discovery Functions

list_benchmarks()

List all available benchmarks registered in the system.

from mcpbr import list_benchmarks

benchmarks = list_benchmarks()
for b in benchmarks:
    print(f"{b['name']:25s} {b['class']}")

Returns: list[dict[str, str]] -- Each dict contains name (the benchmark identifier) and class (the benchmark class name).

!!! example "Sample Output" swe-bench-lite SWEBenchmark swe-bench-verified SWEBenchmark swe-bench-full SWEBenchmark cybergym CyberGymBenchmark humaneval HumanEvalBenchmark mcptoolbench MCPToolBenchmark gsm8k GSM8KBenchmark ...

list_providers()

List all supported model providers.

from mcpbr import list_providers

providers = list_providers()
print(providers)
# ['anthropic', 'openai', 'gemini', 'qwen']

Returns: list[str] -- A list of provider name strings.

list_models()

List all supported models with their metadata.

from mcpbr import list_models

models = list_models()
for m in models:
    print(f"{m['id']:35s} {m['provider']:10s} {m['context_window']:>10,}")

Returns: list[dict[str, str]] -- Each dict contains:

get_version()

Get the current mcpbr version string.

from mcpbr import get_version

version = get_version()
print(version)  # e.g., "0.8.0"

Returns: str -- The version string.

Error Handling

The SDK raises standard Python exceptions:

!!! example "Error Handling Pattern" ```python from mcpbr import MCPBenchmark

try:
    bench = MCPBenchmark("nonexistent.yaml")
except FileNotFoundError as e:
    print(f"Config file not found: {e}")

try:
    bench = MCPBenchmark({"benchmark": "invalid-benchmark"})
except ValueError as e:
    print(f"Invalid configuration: {e}")
```

Testing with the SDK

The SDK is designed to be easily mockable for testing:

import asyncio
from unittest.mock import AsyncMock
from mcpbr import MCPBenchmark, BenchmarkResult

# Create a benchmark instance
bench = MCPBenchmark({
    "mcp_server": {"command": "test-server", "args": []},
    "benchmark": "humaneval",
    "model": "sonnet",
})

# Mock the internal _execute method
bench._execute = AsyncMock(return_value=BenchmarkResult(
    success=True,
    summary={"pass_rate": 0.90},
    tasks=[],
    metadata={},
    total_cost=0.50,
    total_tokens=10000,
    duration_seconds=60.0,
))

# Run the benchmark (uses the mock)
result = asyncio.run(bench.run())
assert result.success
assert result.total_cost == 0.50