Configuration Reference

Configuration Reference

The mcpbr.config module provides Pydantic-based configuration models for the evaluation harness. All configuration can be specified via YAML files or constructed programmatically.

HarnessConfig

The main configuration class for the evaluation harness. Every field has a sensible default except mcp_server (required in single-server mode).

::: mcpbr.config.HarnessConfig options: show_root_heading: true show_source: false

Fields

MCP Server Configuration

Model and Provider

Benchmark Settings

Runtime Settings

Caching and Checkpointing

Docker and Infrastructure

Output

Validators

HarnessConfig includes automatic validation:

• provider: Must be one of VALID_PROVIDERS
• agent_harness: Must be one of VALID_HARNESSES
• benchmark: Must be one of VALID_BENCHMARKS
• timeout_seconds: Must be at least 30
• max_concurrent: Must be at least 1
• budget: Must be positive if set
• thinking_budget: Must be between 1024 and 31999 if set
• Server config consistency: comparison_mode requires both mcp_server_a and mcp_server_b; single mode requires mcp_server

MCPServerConfig

Configuration for an MCP server process.

::: mcpbr.config.MCPServerConfig options: show_root_heading: true show_source: false members: - name - command - args - env - startup_timeout_ms - tool_timeout_ms - setup_command - setup_timeout_ms - get_args_for_workdir - get_setup_command_for_workdir - get_expanded_env

Fields

Methods

get_args_for_workdir(workdir: str) -> list[str]

Replace {workdir} placeholder in args with the actual working directory path.

server = MCPServerConfig(
    command="npx",
    args=["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"],
)
resolved_args = server.get_args_for_workdir("/tmp/task-repo")
# ["-y", "@modelcontextprotocol/server-filesystem", "/tmp/task-repo"]

get_setup_command_for_workdir(workdir: str) -> str | None

Replace {workdir} placeholder in setup_command.

server = MCPServerConfig(
    command="my-server",
    setup_command="cd {workdir} && pip install -e .",
)
cmd = server.get_setup_command_for_workdir("/tmp/task-repo")
# "cd /tmp/task-repo && pip install -e ."

get_expanded_env() -> dict[str, str]

Expand ${VAR} references in env values using os.environ.

import os
os.environ["MY_API_KEY"] = "sk-123"

server = MCPServerConfig(
    command="my-server",
    env={"API_KEY": "${MY_API_KEY}"},
)
expanded = server.get_expanded_env()
# {"API_KEY": "sk-123"}

AzureConfig

Configuration for Azure cloud infrastructure.

::: mcpbr.config.AzureConfig options: show_root_heading: true show_source: false

Fields

InfrastructureConfig

Configuration for the infrastructure mode selector.

::: mcpbr.config.InfrastructureConfig options: show_root_heading: true show_source: false

Fields

!!! warning "Azure Validation" When mode is set to "azure", the azure field is required. Omitting it raises a ValueError.

load_config()

Load configuration from a YAML file with environment variable expansion and inheritance support.

::: mcpbr.config.load_config options: show_root_heading: true show_source: false

Signature

def load_config(
    config_path: str | Path,
    warn_security: bool = True,
) -> HarnessConfig

Features

• Automatically loads .env file from the current directory
• Supports ${VAR} and ${VAR:-default} syntax for environment variables
• Supports config inheritance via the extends field
• Validates all fields with Pydantic
• Warns about hardcoded secrets when warn_security=True

Example

from mcpbr.config import load_config

# Load from YAML file
config = load_config("mcpbr.yaml")

# Suppress security warnings (e.g., in CI)
config = load_config("mcpbr.yaml", warn_security=False)

Exceptions

YAML Configuration Format

Minimal Configuration

mcp_server:
  command: npx
  args:
    - "-y"
    - "@modelcontextprotocol/server-filesystem"
    - "{workdir}"

benchmark: humaneval
model: sonnet

Full Configuration

# MCP Server
mcp_server:
  name: my-mcp-server
  command: npx
  args:
    - "-y"
    - "@modelcontextprotocol/server-filesystem"
    - "{workdir}"
  env:
    API_KEY: "${MY_API_KEY}"
  startup_timeout_ms: 60000
  tool_timeout_ms: 900000
  setup_command: "cd {workdir} && npm install"
  setup_timeout_ms: 300000

# Model and Provider
provider: anthropic
model: sonnet
agent_harness: claude-code
agent_prompt: |
  You are a software engineer. Fix the following bug:
  {problem_statement}

# Benchmark
benchmark: swe-bench-verified
sample_size: 20
filter_category:
  - django
  - flask

# Runtime
timeout_seconds: 600
max_concurrent: 4
max_iterations: 15
thinking_budget: 10000
budget: 50.0

# Error Handling
continue_on_error: true
max_failures: 10

# Caching
cache_enabled: true
cache_dir: ~/.cache/mcpbr

# Checkpointing
checkpoint_interval: 5

# Docker
use_prebuilt_images: true
volumes:
  /host/cache: /container/cache

# Output
output_dir: ./results
disable_logs: false
enable_profiling: true

Comparison Mode Configuration

comparison_mode: true

mcp_server_a:
  name: server-v1
  command: npx
  args: ["-y", "mcp-server-v1", "{workdir}"]

mcp_server_b:
  name: server-v2
  command: npx
  args: ["-y", "mcp-server-v2", "{workdir}"]

benchmark: swe-bench-verified
model: sonnet
sample_size: 50

Azure Infrastructure Configuration

mcp_server:
  command: npx
  args: ["-y", "@modelcontextprotocol/server-filesystem", "{workdir}"]

benchmark: swe-bench-verified
model: sonnet
sample_size: 100

infrastructure:
  mode: azure
  azure:
    resource_group: mcpbr-eval-rg
    location: eastus2
    vm_size: Standard_D8s_v3
    disk_gb: 500
    auto_shutdown: true
    preserve_on_error: true
    env_keys_to_export:
      - ANTHROPIC_API_KEY
      - GITHUB_TOKEN
    python_version: "3.11"

Config Inheritance

Configs can extend other configs using the extends field:

# base.yaml
mcp_server:
  command: npx
  args: ["-y", "my-server", "{workdir}"]
model: sonnet
timeout_seconds: 300

# production.yaml
extends: base.yaml
benchmark: swe-bench-verified
sample_size: 500
budget: 100.0

Environment Variable Support

Syntax

.env File

mcpbr automatically loads a .env file from the current directory:

# .env
ANTHROPIC_API_KEY=sk-ant-...
MY_CUSTOM_VAR=some-value

Usage in YAML

mcp_server:
  command: my-server
  env:
    API_KEY: "${ANTHROPIC_API_KEY}"
    DEBUG: "${DEBUG:-false}"
    REGION: "${AWS_REGION:-us-east-1}"

!!! warning "Security" Do not hardcode API keys or secrets directly in YAML config files. Use environment variables or a .env file instead. mcpbr will warn about potential hardcoded secrets when warn_security=True (the default).

Programmatic Configuration

Creating a Default Config

from mcpbr.config import create_default_config

config = create_default_config()
# Uses default MCP server (filesystem), anthropic provider, sonnet model

Building Config Programmatically

from mcpbr.config import HarnessConfig, MCPServerConfig, InfrastructureConfig, AzureConfig

config = HarnessConfig(
    mcp_server=MCPServerConfig(
        name="my-server",
        command="uvx",
        args=["my-mcp-server", "--workdir", "{workdir}"],
        env={"API_KEY": "from-env"},
    ),
    provider="anthropic",
    model="sonnet",
    benchmark="swe-bench-verified",
    sample_size=10,
    timeout_seconds=600,
    max_concurrent=2,
    budget=25.0,
    infrastructure=InfrastructureConfig(
        mode="azure",
        azure=AzureConfig(
            resource_group="my-eval-rg",
            location="eastus2",
            cpu_cores=8,
            memory_gb=32,
        ),
    ),
)