Tool Star Eval

tool-star-eval

Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning — Dong et al. (2025) (arXiv:2505.16410, 2025)

What this evaluates

Evaluates an LLM's ability to autonomously invoke and coordinate multiple tools (search, code, calculator, etc.) for complex reasoning tasks. It probes both computational reasoning (math) and knowledge-intensive reasoning (open-domain QA) under a multi-tool collaborative setting.

Datasets

• AIME2024 — total ?; splits: test (-1)
• AIME2025 — total ?; splits: test (-1)
• MATH500 — total ?; splits: test (-1)
• MATH — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• GAIA — total ?; splits: test (-1)
• HLE — total ?; splits: test (-1)
• WebWalker — total ?; splits: test (-1)
• HotpotQA — total ?; splits: test (-1)
• 2WikiMultihopQA — total ?; splits: test (-1)
• Musique — total ?; splits: test (-1)
• Bamboogle — total ?; splits: test (-1)

Metrics

• LLM-based judging accuracy (primary) — range: percent
  • Correctness is determined by an LLM judge (Qwen2.5-72B-Instruct) rather than exact string matching. The judge evaluates whether the model's final answer matches the ground truth.
• token-level F1 score — range: percent
  • Standard token-level F1 score computed between the extracted answer and the ground truth for open-domain QA tasks.
• Tool-use Efficiency ($T_E$) — range: percent
  • $T_E = \frac{1}{N}\sum_{i=1}^{N}\frac{S_i}{T^c_i}$, where N is the number of datasets, $S_i$ is the number of correct answers, and $T^c_i$ is the total number of samples for the i-th dataset when using tools.

Input / output format

Input: A reasoning question or prompt requiring multi-step problem solving.

Output: Step-by-step reasoning with tool invocations, followed by a final answer enclosed in \box{} tags.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'computational':
        return llm_judge_accuracy(predictions, golds, judge='Qwen2.5-72B-Instruct')
    elif task_type == 'qa':
        return token_f1_score(predictions, golds)
    # Tool-use Efficiency across N datasets
    N = len(datasets)
    efficiency_sum = 0
    for i in range(N):
        S_i = count_correct(predictions[i], golds[i])
        T_c_i = len(golds[i])
        efficiency_sum += S_i / T_c_i
    return efficiency_sum / N

Common pitfalls

• Answers must be enclosed in \box{} tags for extraction; missing tags cause evaluation failure.
• Computational tasks use a specific LLM judge (Qwen2.5-72B-Instruct), not exact match.
• Tool-use Efficiency aggregates accuracy across datasets, which can obscure per-task performance variations.

Evidence (verbatim from paper)

For all tasks, we follow previous work*[[31]]* and extract answers from the model output enclosed in \box{}. For computational reasoning tasks, we follow Search-o1*[[31]]* employ LLM-based judging (Qwen2.5-72B=Instruct) to ensure answer correctness. For open-domain QA tasks, we adopt token-level F1 score as the evaluation metric. To assess tool usage efficiency, we propose the Tool-use Efficiency metric: $T_{E}=\frac{1}{N}\sum_{i=1}^{N}\frac{S_{i}}{T^{c}{i}}$ where $N$ is the number of datasets, $S{i}$ and $T^{c}_{i}$ denote the number of correct answers and the number of total samples for the $i$-th dataset when using tools.

Citation

@misc{dong2025toolstar,
  title={Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning},
  author={Dong et al. (2025)},
  year={2025},
  note={arXiv:2505.16410}
}

• arXiv: 2505.16410