Comprehensive Statistics

Comprehensive Statistics

MCPBR now provides detailed statistics beyond basic success rates to help you understand and optimize your MCP server performance.

What's Included

The enhanced statistics provide insights across multiple dimensions:

1. Token Usage Statistics

• Total tokens (input/output) across all tasks
• Per-task averages and ranges (min/max)
• Detailed breakdown for both MCP and baseline runs
• Helps you understand: API usage patterns, cost drivers, and efficiency

2. Cost Breakdown

• Total cost for the evaluation run
• Average cost per task and per resolved task
• Min/max costs to identify outliers
• Cost-effectiveness metrics comparing MCP vs baseline

3. Tool Usage Statistics (MCP Only)

• Total tool calls with success/failure rates
• Per-tool breakdown showing reliability
• Most used and most failed tools
• Unique tools used and average calls per task
• Helps you understand: Which tools are essential, which need improvement

4. Error Analysis

• Total errors and error rates
• Categorization (timeout, network, docker, MCP, etc.)
• Most common errors with sample instances
• Timeout tracking separately from other errors
• Helps you understand: Common failure modes, infrastructure issues

5. Iteration Statistics

• Average iterations per task
• Distribution showing iteration patterns
• Min/max iterations to find edge cases
• Helps you understand: Task complexity, convergence patterns

Viewing Statistics

Console Output

When you run an evaluation, comprehensive statistics are automatically displayed:

mcpbr run config.yaml

You'll see Rich-formatted tables showing:

Token Usage Statistics
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Metric          ┃ MCP Agent  ┃ Baseline ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Input     │ 45,230     │ 38,120   │
│ Total Output    │ 123,450    │ 98,340   │
│ Avg Input/Task  │ 4,523      │ 3,812    │
│ Avg Output/Task │ 12,345     │ 9,834    │
└─────────────────┴────────────┴──────────┘

Iteration Statistics
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Metric             ┃ MCP Agent  ┃ Baseline ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Iterations   │ 87         │ 56       │
│ Avg Iterations/Task│ 8.7        │ 5.6      │
│ Max Iterations     │ 15         │ 10       │
└────────────────────┴────────────┴──────────┘

MCP Tool Usage Statistics
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Metric           ┃ Value       ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Calls      │ 1,247       │
│ Successful Calls │ 1,198       │
│ Failed Calls     │ 49          │
│ Failure Rate     │ 3.9%        │
│ Unique Tools Used│ 12          │
└──────────────────┴─────────────┘

Top 10 Most Used Tools:
┏━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Tool       ┃ Calls ┃ Success Rate ┃
┡━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━┩
│ Read       │ 456   │ 99.3%        │
│ Bash       │ 312   │ 95.2%        │
│ Write      │ 234   │ 98.7%        │
└────────────┴───────┴──────────────┘

Error Analysis
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Metric       ┃ MCP Agent  ┃ Baseline ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Errors │ 2          │ 3        │
│ Error Rate   │ 20.0%      │ 30.0%    │
│ Timeout Count│ 1          │ 2        │
└──────────────┴────────────┴──────────┘

JSON Output

All statistics are included in JSON results:

mcpbr run config.yaml --format json -o results.json

{
  "summary": {
    "comprehensive_stats": {
      "mcp_tokens": {
        "total_input": 45230,
        "total_output": 123450,
        "avg_input_per_task": 4523.0,
        "avg_output_per_task": 12345.0,
        "max_input_per_task": 8920,
        "min_input_per_task": 2340,
        "per_task": {
          "task-1": {
            "input": 4523,
            "output": 12345,
            "total": 16868
          }
        }
      },
      "mcp_costs": {
        "total_cost": 0.5234,
        "avg_cost_per_task": 0.0523,
        "max_cost_per_task": 0.0892,
        "min_cost_per_task": 0.0234,
        "cost_per_resolved": 0.0654,
        "per_task": {
          "task-1": 0.0523
        }
      },
      "mcp_tools": {
        "total_calls": 1247,
        "total_successes": 1198,
        "total_failures": 49,
        "failure_rate": 0.0393,
        "unique_tools_used": 12,
        "avg_calls_per_task": 124.7,
        "per_tool": {
          "Read": {
            "total": 456,
            "succeeded": 453,
            "failed": 3,
            "failure_rate": 0.0066
          }
        },
        "most_used_tools": {
          "Read": 456,
          "Bash": 312
        },
        "most_failed_tools": {
          "Docker": 20,
          "MCP_Tool": 15
        }
      },
      "mcp_errors": {
        "total_errors": 2,
        "error_rate": 0.2,
        "timeout_count": 1,
        "timeout_rate": 0.1,
        "error_categories": {
          "timeout": 1,
          "docker": 1
        },
        "most_common_errors": {
          "Timeout exceeded": 1,
          "Docker container failed": 1
        },
        "sample_errors": [
          {
            "instance_id": "task-5",
            "error": "Timeout exceeded",
            "category": "timeout"
          }
        ]
      },
      "mcp_iterations": {
        "total_iterations": 87,
        "avg_iterations": 8.7,
        "max_iterations": 15,
        "min_iterations": 3,
        "distribution": {
          "3": 1,
          "5": 2,
          "8": 4,
          "15": 1
        },
        "per_task": {
          "task-1": 8
        }
      }
    }
  }
}

Markdown Report

Statistics are also included in the Markdown report:

mcpbr run config.yaml --format markdown -o report.md

The report includes:

• Summary tables comparing MCP vs Baseline
• Detailed token usage breakdown
• Cost analysis with per-task details
• Tool usage statistics with failure analysis
• Error analysis with categorization
• Iteration distribution charts

Interpreting the Statistics

Token Usage

• High input tokens: May indicate verbose prompts or large context
• High output tokens: Could suggest the model is being too verbose
• Wide variance: Tasks have different complexity levels
• Action: Consider optimizing prompts for high-token outliers

Tool Usage

• High failure rate (>10%): Infrastructure or MCP server issues
• Unused tools: May indicate poor tool discoverability
• Tool concentration: Most tasks use a few core tools
• Action: Focus on improving reliability of most-used tools

Errors

• High timeout rate: Tasks too complex or timeout too short
• Network errors: Infrastructure reliability issues
• Docker errors: Environment setup problems
• MCP errors: Server-specific issues
• Action: Address most common error categories first

Iterations

• High average iterations: Tasks require multiple attempts
• Wide distribution: Some tasks are much harder than others
• Max iterations hit: May need to increase iteration limit
• Action: Analyze high-iteration tasks for common patterns

Use Cases

1. Debugging MCP Server Issues

# Look at tool failure rates
mcpbr run config.yaml | grep "Failed Calls"

# Check error categories
mcpbr run config.yaml --format json -o results.json
cat results.json | jq '.summary.comprehensive_stats.mcp_errors.error_categories'

2. Cost Optimization

# Find expensive tasks
cat results.json | jq '.summary.comprehensive_stats.mcp_costs.per_task | to_entries | sort_by(.value) | reverse | .[0:5]'

# Compare MCP vs baseline costs
cat results.json | jq '{
  mcp: .summary.comprehensive_stats.mcp_costs.total_cost,
  baseline: .summary.comprehensive_stats.baseline_costs.total_cost,
  difference: (.summary.comprehensive_stats.mcp_costs.total_cost - .summary.comprehensive_stats.baseline_costs.total_cost)
}'

3. Performance Analysis

# Check iteration patterns
cat results.json | jq '.summary.comprehensive_stats.mcp_iterations.distribution'

# Find high-iteration outliers
cat results.json | jq '.summary.comprehensive_stats.mcp_iterations.per_task | to_entries | sort_by(.value) | reverse | .[0:5]'

4. Tool Reliability Monitoring

# List tools with high failure rates
cat results.json | jq '.summary.comprehensive_stats.mcp_tools.per_tool | to_entries | map(select(.value.failure_rate > 0.1)) | sort_by(.value.failure_rate) | reverse'

# Track improvement over time
for file in results-*.json; do
  echo "$file: $(jq '.summary.comprehensive_stats.mcp_tools.failure_rate' $file)"
done

API Usage

If you're using MCPBR programmatically, you can access statistics directly:

from mcpbr.harness import run_evaluation
from mcpbr.statistics import calculate_comprehensive_statistics

# Run evaluation
results = await run_evaluation(config)

# Statistics are automatically included
stats_dict = results.summary["comprehensive_stats"]

# Or calculate manually from results
from mcpbr.statistics import ComprehensiveStatistics
stats = ComprehensiveStatistics(**stats_dict)

# Access specific metrics
print(f"MCP tool failure rate: {stats.mcp_tools.failure_rate:.1%}")
print(f"Average iterations: {stats.mcp_iterations.avg_iterations:.1f}")
print(f"Total cost: ${stats.mcp_costs.total_cost:.4f}")

# Export to dict for custom processing
stats_dict = stats.to_dict()

Best Practices

1. Regular Monitoring: Track statistics across runs to identify trends
2. Set Baselines: Establish expected ranges for key metrics
3. Alert on Anomalies: High failure rates or costs should trigger investigation
4. Iterate on Tools: Use statistics to prioritize tool improvements
5. Cost Budgets: Monitor costs against budgets using cost statistics
6. Error Analysis: Categorize and track error patterns over time

See Also

• Configuration Guide
• MCP Integration
• Best Practices