Livemcpbench Eval

livemcpbench-eval

LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? — Mo Guozhao et al. (2025) (arXiv:2508.01780, 2025)

What this evaluates

Evaluates LLM agents' capability to dynamically discover, select, and chain Model Context Protocol (MCP) tools to complete complex, multi-step real-world tasks. It probes meta-tool learning and multi-tool collaboration in large-scale, time-varying tool ecosystems.

Datasets

• LiveMCPBench — total 95; splits: test (95)

Metrics

• task success rate (primary) — range: percent
  • The percentage of tasks out of the total 95 that the agent successfully completes within the MCP environment. Assessed via LiveMCPEval, an LLM-as-a-Judge framework (using Deepseek-V3) that evaluates multi-step tool invocation trajectories against task requirements. Calculated as (number of tasks judged successful / 95) * 100.

Input / output format

Input: Natural language task prompts requiring interaction with a dynamic pool of 527 tools across 70 MCP servers.

Output: Sequential tool invocation trajectories (ReACT-style) and a final binary success/failure status for each task.

Scoring recipe

def compute_task_success_rate(predictions, gold):
    successful = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (successful / len(gold)) * 100

Common pitfalls

• Using advanced reasoning models (e.g., Deepseek-R1, Claude-Opus-4) as the LLM-as-a-Judge evaluator yields lower human agreement rates (60-70%) due to difficulties processing long execution trajectories.
• The benchmark evaluates dynamic, real-world MCP servers which may change over time, making exact reproducibility dependent on server availability and the specific evaluation timestamp.

Evidence (verbatim from paper)

We show the task success rates for different models in Table[2]. We can see that: 1. 1. Meta-Tool-Learning Capabilities in Claude Models. The Claude series demonstrates remarkable meta-tool-learning proficiency, with Claude-Sonnet-4 and Claude-Opus-4 achieving success rates of 78.95% and 70.53% respectively.

Citation

@misc{mo2025livemcpbench,
  title={LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?},
  author={Mo Guozhao et al. (2025)},
  year={2025},
  note={arXiv:2508.01780}
}

• arXiv: 2508.01780