qhjqhj00 toolmind-eval
ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset — Chen Yang et al. (arXiv:2511.15718, 2025)
What this evaluates
Evaluates large language models' tool-use and function-calling capabilities, specifically probing multi-turn dialogues and agentic workflows such as search and memory retrieval.
Datasets
- BFCL-v4 — total ?; splits: test (-1)
- τ-Bench — total ?; splits: test (-1)
- τ²-Bench — total ?; splits: test (-1)
Metrics
BFCL-v4 Overall(primary) — range: percent- Percentage of correctly executed tool calls or environment interactions, averaged across single-turn, multi-turn, and agentic sub-tasks.
τ-Bench Avg.— range: percent- Average accuracy across retail and airline domains, measured via environment interaction with a GPT-4o user simulator.
τ²-Bench Avg.— range: percent- Average accuracy across retail and airline domains in a multi-turn agentic setting, evaluated with a GPT-4o user simulator.
Input / output format
Input: Multi-turn conversation history containing user queries, prior tool calls, and function schemas.
Output: Model-generated reasoning traces, content responses, and structured tool calls.
Scoring recipe
def compute_metric(predictions, gold):
correct = 0
for pred, gold_inst in zip(predictions, gold):
if execute_tool_call(pred) == gold_inst:
correct += 1
return (correct / len(gold)) * 100
Common pitfalls
- Single-turn scores may fluctuate modestly while multi-turn and agentic scores show significant gains.
- BFCL-v4 distinguishes between non-live (static AST) and live (dynamic execution) evaluation, which can yield different results.
- τ-bench and τ²-bench rely on a GPT-4o user simulator, introducing potential variability in interaction dynamics.
Evidence (verbatim from paper)
Overall Performance. The performance of our SFT models on τ-Bench [20], and τ²-bench [1], BFCL-v4 [11] is summarized in Table 2 and Table 3. On BFCL-v4, we observe notable improvements in both Qwen3-8B and Qwen3-14B after training. Although their single-turn evaluation scores show modest fluctuations, they remain above most baseline models. Performance gains are most pronounced in multi-turn and agentic evaluations.
Citation
@misc{yang2025toolmind,
title={ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset},
author={Chen Yang et al.},
year={2025},
note={arXiv:2511.15718}
}
- arXiv: 2511.15718