Benchmarks API Reference

Benchmarks API Reference

The mcpbr.benchmarks package defines the benchmark abstraction layer and provides implementations for 29 benchmarks. Use the Benchmark protocol to add custom benchmarks or interact with existing ones programmatically.

from mcpbr.benchmarks import (
    Benchmark,
    BenchmarkTask,
    BENCHMARK_REGISTRY,
    create_benchmark,
    list_benchmarks,
)

Benchmark Protocol

The Benchmark protocol defines the interface that all benchmark implementations must satisfy. It is decorated with @runtime_checkable, so you can use isinstance() checks.

::: mcpbr.benchmarks.base.Benchmark options: show_root_heading: true show_source: false

Required Attributes

Required Methods

load_tasks()

Load tasks from the benchmark dataset with optional filtering.

def load_tasks(
    self,
    sample_size: int | None = None,
    task_ids: list[str] | None = None,
    level: int | None = None,
    filter_difficulty: list[str] | None = None,
    filter_category: list[str] | None = None,
    filter_tags: list[str] | None = None,
) -> list[dict[str, Any]]

Returns: list[dict[str, Any]] -- Task dictionaries in benchmark-specific format.

normalize_task()

Convert a benchmark-specific task dictionary to the normalized BenchmarkTask format.

def normalize_task(self, task: dict[str, Any]) -> BenchmarkTask

Returns: BenchmarkTask with standardized fields.

create_environment()

Create an isolated Docker environment for the task.

async def create_environment(
    self,
    task: dict[str, Any],
    docker_manager: DockerEnvironmentManager,
) -> TaskEnvironment

Returns: TaskEnvironment with the Docker container and working directory.

evaluate()

Evaluate a solution (e.g., a patch or generated code) against the task.

async def evaluate(
    self,
    env: TaskEnvironment,
    task: dict[str, Any],
    solution: str,
) -> dict[str, Any]

Returns: Dictionary with evaluation results including a resolved boolean.

get_prebuilt_image()

Get the pre-built Docker image name for a task, if available.

def get_prebuilt_image(self, task: dict[str, Any]) -> str | None

Returns: Docker image name or None.

get_prompt_template()

Get the benchmark-specific prompt template for agents.

def get_prompt_template(self) -> str

Returns: Prompt template string with {problem_statement} placeholder.

BenchmarkTask

Normalized task representation across different benchmarks.

::: mcpbr.benchmarks.base.BenchmarkTask options: show_root_heading: true show_source: false

Fields

!!! example "BenchmarkTask Example" ```python from mcpbr.benchmarks import BenchmarkTask

task = BenchmarkTask(
    task_id="django__django-11099",
    problem_statement="Fix the bug in QuerySet.union() that drops ORDER BY...",
    repo="django/django",
    commit="abc123def456",
    metadata={
        "difficulty": "medium",
        "fail_to_pass": ["tests.queries.test_qs_combinators.QuerySetSetOperationTests"],
    },
)
```

TaskEnvironment

The Docker-based execution environment for benchmark tasks. This is returned by create_environment() and provides methods for interacting with the container.

Key Properties

Key Methods

create_benchmark()

Factory function to create benchmark instances from the registry.

from mcpbr.benchmarks import create_benchmark

# Create a benchmark by name
benchmark = create_benchmark("humaneval")

# SWE-bench variants auto-set the dataset
benchmark = create_benchmark("swe-bench-verified")
# Internally sets dataset="SWE-bench/SWE-bench_Verified"

# Pass additional kwargs to the constructor
benchmark = create_benchmark("cybergym", level=2)

Raises: ValueError if the benchmark name is not recognized.

BENCHMARK_REGISTRY

Dictionary mapping benchmark IDs to their implementation classes.

from mcpbr.benchmarks import BENCHMARK_REGISTRY

# List all registered benchmarks
for name, cls in BENCHMARK_REGISTRY.items():
    print(f"{name:25s} -> {cls.__name__}")

Available Benchmarks

mcpbr ships with 29 benchmark implementations:

Software Engineering

Code Generation

Reasoning and Knowledge

Agent and Tool Use

Security

Custom

Implementing a Custom Benchmark

To add a new benchmark, implement the Benchmark protocol:

from typing import Any
from mcpbr.benchmarks.base import Benchmark, BenchmarkTask
from mcpbr.docker_env import DockerEnvironmentManager, TaskEnvironment


class MyBenchmark:
    """Custom benchmark implementation."""

    name: str = "my-benchmark"

    def load_tasks(
        self,
        sample_size: int | None = None,
        task_ids: list[str] | None = None,
        level: int | None = None,
        filter_difficulty: list[str] | None = None,
        filter_category: list[str] | None = None,
        filter_tags: list[str] | None = None,
    ) -> list[dict[str, Any]]:
        """Load tasks from your dataset."""
        tasks = [
            {
                "instance_id": "task-001",
                "problem_statement": "Implement a function that...",
                "repo": "my-org/my-repo",
                "base_commit": "abc123",
                "difficulty": "easy",
            },
            # ... more tasks
        ]

        # Apply filters
        if task_ids:
            tasks = [t for t in tasks if t["instance_id"] in task_ids]
        if filter_difficulty:
            tasks = [t for t in tasks if t.get("difficulty") in filter_difficulty]
        if sample_size:
            tasks = tasks[:sample_size]

        return tasks

    def normalize_task(self, task: dict[str, Any]) -> BenchmarkTask:
        """Convert to normalized format."""
        return BenchmarkTask(
            task_id=task["instance_id"],
            problem_statement=task["problem_statement"],
            repo=task["repo"],
            commit=task["base_commit"],
            metadata={"difficulty": task.get("difficulty")},
        )

    async def create_environment(
        self,
        task: dict[str, Any],
        docker_manager: DockerEnvironmentManager,
    ) -> TaskEnvironment:
        """Create a Docker environment for the task."""
        image = self.get_prebuilt_image(task)
        return await docker_manager.create_environment(
            image=image or "python:3.11-slim",
            instance_id=task["instance_id"],
            repo=task["repo"],
            commit=task["base_commit"],
        )

    async def evaluate(
        self,
        env: TaskEnvironment,
        task: dict[str, Any],
        solution: str,
    ) -> dict[str, Any]:
        """Evaluate the agent's solution."""
        # Apply the solution and run tests
        await env.write_file("/tmp/solution.py", solution)
        exit_code, output = await env.exec_command(
            "cd /workspace && python -m pytest tests/ -v"
        )
        return {
            "resolved": exit_code == 0,
            "test_output": output,
        }

    def get_prebuilt_image(self, task: dict[str, Any]) -> str | None:
        """Return pre-built Docker image if available."""
        return None  # No pre-built images

    def get_prompt_template(self) -> str:
        """Return the prompt template for agents."""
        return (
            "You are a software engineer. Solve the following problem:\n\n"
            "{problem_statement}\n\n"
            "Provide your solution as a Python implementation."
        )

Registering Your Benchmark

To make your benchmark available via create_benchmark() and the CLI, register it in BENCHMARK_REGISTRY:

# In mcpbr/benchmarks/__init__.py
from .my_benchmark import MyBenchmark

BENCHMARK_REGISTRY["my-benchmark"] = MyBenchmark

And add the benchmark ID to VALID_BENCHMARKS in mcpbr/config.py:

VALID_BENCHMARKS = (
    # ... existing benchmarks ...
    "my-benchmark",
)

!!! tip "Protocol Compliance" You can verify your implementation satisfies the protocol at runtime: ```python from mcpbr.benchmarks import Benchmark

my_bench = MyBenchmark()
assert isinstance(my_bench, Benchmark)  # Runtime check
```

list_benchmarks()

List all available benchmark names from the registry.

from mcpbr.benchmarks import list_benchmarks

names = list_benchmarks()
print(names)
# ['swe-bench-lite', 'swe-bench-verified', 'swe-bench-full', 'cybergym',
#  'humaneval', 'mcptoolbench', 'gsm8k', 'mbpp', 'math', ...]

Returns: list[str] -- Sorted list of benchmark identifier strings.