Toolsandbox Eval

toolsandbox-eval

ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities — Lu et al. (2024) (arXiv:2408.04682, 2024)

What this evaluates

Evaluates LLM tool-use capabilities in a stateful, conversational, and interactive setting. It probes the model's ability to handle implicit state dependencies, canonicalize arguments, handle insufficient information, and maintain efficiency across single/multiple tool calls and user turns.

Datasets

• ToolSandbox — total ?; splits: test (-1); repo https://github.com/apple/ToolSandbox

Metrics

• average similarity score (primary) — range: percent
  • The average similarity score between the model's generated tool calls/responses and the gold/reference trajectory, computed per instance and averaged across all instances or specific scenario categories. Scores are reported on a 0-100 scale.

Input / output format

Input: Conversational user queries presented in an interactive environment with access to a predefined set of tools. The model receives a minimalist system prompt and must generate tool calls or text responses based on the current conversation state and tool availability.

Output: Sequential tool calls and conversational responses generated by the LLM within an interactive loop, subject to an execution environment that tracks state and returns tool outputs.

Scoring recipe

scores = []
for instance in test_set:
    trajectory = run_model_interactively(instance, prompt=minimalist_prompt)
    sim = compute_similarity(trajectory, gold_trajectory)
    scores.append(sim)
avg_score = sum(scores) / len(scores)
# Report avg_score overall and grouped by scenario categories (STC, MTC, SD, etc.) and augmentations (DT, TNS, etc.)

Common pitfalls

• Models may hallucinate tool names or arguments when provided tools are insufficient, artificially lowering the 'Insufficient Information' score.
• Larger models often issue parallel tool calls for state-dependent tasks, causing race conditions in the execution environment and degrading performance.
• Using complex prompt engineering can inflate scores; the benchmark intentionally uses a minimalist prompt to measure innate tool-use capability.

Evidence (verbatim from paper)

When evaluating the models, all models use the same minimalist prompt shown in Figure [8] for comparison fairness. We do not include additional prompt engineering for the models, as we consider prompt engineering gains orthogonal to the innate model capability surfaced by simpler prompting. Table [5] shows the average similarity for each of the scenario categories described in Section [3].

Citation

@misc{lu2024toolsandbox,
  title={ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities},
  author={Lu et al. (2024)},
  year={2024},
  note={arXiv:2408.04682}
}

• arXiv: 2408.04682