097 Index C5c1c42d

API Reference

Comprehensive reference documentation for the mcpbr Python API. Use these modules to programmatically configure, execute, and analyze MCP server benchmarks.

Quick Start

The fastest way to use mcpbr programmatically is through the SDK module:

from mcpbr import MCPBenchmark, list_benchmarks, list_models, get_version

# Check available benchmarks and models
print(get_version())  # e.g., "0.8.0"
for b in list_benchmarks():
    print(b["name"], b["class"])

# Configure and validate a benchmark
bench = MCPBenchmark({
    "mcp_server": {
        "command": "npx",
        "args": ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"],
    },
    "benchmark": "humaneval",
    "model": "sonnet",
    "sample_size": 10,
})

is_valid, errors = bench.validate()
if is_valid:
    plan = bench.dry_run()
    print(plan)

!!! tip "SDK vs Harness" The SDK (mcpbr.sdk) provides a high-level, user-friendly interface for configuration and validation. For full evaluation execution with Docker environments, use the Harness (mcpbr.harness.run_evaluation) directly.

Core Modules

mcpbr is organized into several focused modules. Click through to each sub-page for detailed API documentation with examples.

Architecture Overview

mcpbr
 |-- sdk.py                  # Public Python SDK (MCPBenchmark, list_*)
 |-- config.py               # Configuration models (HarnessConfig, MCPServerConfig)
 |-- harness.py              # Evaluation orchestration (run_evaluation)
 |-- models.py               # Model registry (SUPPORTED_MODELS)
 |-- benchmarks/
 |   |-- base.py             # Benchmark protocol and BenchmarkTask
 |   |-- swebench.py         # SWE-bench implementation
 |   |-- humaneval.py        # HumanEval implementation
 |   +-- ...                 # 27+ benchmark implementations
 |-- analytics/
 |   |-- database.py         # SQLite results storage
 |   |-- statistical.py      # Hypothesis testing
 |   |-- comparison.py       # Multi-model comparison
 |   |-- regression_detector.py
 |   |-- ab_testing.py       # A/B testing framework
 |   |-- leaderboard.py      # Rankings generation
 |   |-- metrics.py          # Custom metrics registry
 |   |-- trends.py           # Time-series trends
 |   |-- anomaly.py          # Outlier detection
 |   |-- correlation.py      # Metric correlations
 |   |-- error_analysis.py   # Error clustering
 |   +-- difficulty.py       # Task difficulty scoring
 +-- reports/
     |-- html_report.py      # Interactive HTML reports
     |-- enhanced_markdown.py # GitHub-flavored markdown
     +-- pdf_report.py       # Print-friendly PDF reports

Harness API

The harness module orchestrates the full evaluation pipeline, including task loading, Docker environment management, agent execution, and result aggregation.

run_evaluation

::: mcpbr.harness.run_evaluation options: show_root_heading: true show_source: false

EvaluationResults

::: mcpbr.harness.EvaluationResults options: show_root_heading: true show_source: false

TaskResult

::: mcpbr.harness.TaskResult options: show_root_heading: true show_source: false

Models

ModelInfo

::: mcpbr.models.ModelInfo options: show_root_heading: true show_source: false

Model Functions

::: mcpbr.models.list_supported_models options: show_root_heading: true show_source: false

::: mcpbr.models.get_model_info options: show_root_heading: true show_source: false

::: mcpbr.models.is_model_supported options: show_root_heading: true show_source: false

::: mcpbr.models.validate_model options: show_root_heading: true show_source: false

Constants

Default Values

from mcpbr.models import DEFAULT_MODEL
from mcpbr.config import VALID_PROVIDERS, VALID_HARNESSES, VALID_BENCHMARKS

print(DEFAULT_MODEL)       # "sonnet"
print(VALID_PROVIDERS)     # ("anthropic", "openai", "gemini", "qwen")
print(VALID_HARNESSES)     # ("claude-code",)
print(VALID_BENCHMARKS)    # 29 benchmark identifiers

Supported Models