Toolbench Eval

toolbench-eval

On the Tool Manipulation Capability of Open-source Large Language Models — Xu et al. (2023) (arXiv:2305.16504, 2023)

What this evaluates

This benchmark evaluates an LLM's ability to understand API documentation, select the correct tool, and generate valid arguments to fulfill a user's goal. It probes tool manipulation capabilities across a diverse set of 8 real-world applications, ranging from simple single-call tasks to complex multi-step reasoning.

Datasets

• ToolBench — total ?; splits: test (-1); repo https://github.com/sambanova/toolbench

Metrics

• success rate (primary) — range: percent
  • The percentage of user goals for which the model successfully generates a valid tool call that completes the task. Calculated as (number of successful goals / total goals) * 100.

Input / output format

Input: User goal/query, API documentation, and optionally a few-shot in-context demonstration examples.

Output: A structured tool call specification containing the API name and populated arguments.

Scoring recipe

successes = 0
for goal in goals:
    call = parse_output(model.generate(goal, docs, demos))
    result = execute(call)
    if result.matches_ground_truth(goal):
        successes += 1
return (successes / len(goals)) * 100

Common pitfalls

• The Tabletop task is evaluated exclusively in a few-shot setting, unlike other tasks which use zero-shot baselines.
• Results are averaged over 3 runs with different random seeds, but variance is often ignored in the main paper.
• Complex reasoning tasks like Google Sheets and WebShop show high variance and low success rates even after enhancement.

Evidence (verbatim from paper)

We run each job 3 times with different random seeds and report average accuracy. Table 6 exhibits significant disparities in tool manipulation between the closed GPT-4 API and open-source models in the out-of-the-box zero-shot setting. ... the success rates of the open-source LLMs can improve up to 90%.

Citation

@misc{xu2023toolmanipulation,
  title={On the Tool Manipulation Capability of Open-source Large Language Models},
  author={Xu et al. (2023)},
  year={2023},
  note={arXiv:2305.16504}
}

• arXiv: 2305.16504