agentskillexchange Evaluate agent and model workflows with EvalScope
Run repeatable EvalScope benchmark suites for LLM, VLM, RAG, and agent workflows, then inspect traces and reports before changing models or prompts.
Prerequisites
Python 3.10+, pip, evalscope package, model endpoint or local model backend, API credentials when evaluating hosted models, selected benchmark datasets
Installation
Use the upstream install or setup path that matches your environment:
- pip install evalscope
- pip install 'evalscope[service]'
Requirements and caveats from upstream:
- 🤖 Agent Evaluation Mode: Drives benchmarks (e.g. GSM8K, AIME, SWE-bench Agentic) inside a controlled multi-turn AgentLoop with pluggable strategies, tools and Docker sandbox; full per-sample Agent Trace is recorde...
- 🔥 [2026.05.26] Added the GAIA agent benchmark (multi-turn ReAct + bash in a Docker sandbox, official rule-based scorer) and generic [MCP server](...
Basic usage or getting-started notes:
🔥 [2026.05.19] Added support for SWE-bench_Pro and τ³-bench: S...
🔥 [2026.05.15] Introduced Agent Evaluation Mode: any benchmark based on DefaultDataAdapter (GSM8K, AIME, IFEval, etc.) can now be driven through a multi-turn AgentLoop with pluggable strategies (function_calli...
🔥 [2026.05.08] Partnered with LightSeek to launch TokenSpeed, a speed-of-light LLM inference engine for agentic workloads. EvalScop...
Extracted from upstream docs: https://raw.githubusercontent.com/modelscope/evalscope/HEAD/README.md