Toolalpaca Eval

toolalpaca-eval

ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases — Tang et al. (2023) (arXiv:2306.05301, 2023)

What this evaluates

Evaluates a language model's ability to generalize tool-use capabilities to unseen tools and APIs through multi-turn interaction, parameter selection, and final response generation. It measures how well compact models trained on simulated data can adapt to real-world and out-of-dataset tool scenarios without task-specific fine-tuning.

Datasets

• ToolAlpaca Evaluation Set — total ?; splits: test (-1); repo https://github.com/tangqiaoyu/ToolAlpaca
• GPT4Tools Test Set — total 602; splits: test (602); repo https://github.com/tangqiaoyu/ToolAlpaca

Metrics

• Procedure — range: percent
  • Evaluates the model's proficiency in accurately selecting suitable actions, utilizing correct parameters, and avoiding redundant actions. Scored as a percentage.
• Response — range: percent
  • Measures whether the final response can satisfy the user's instruction. Scored as a percentage.
• Overall (primary) — range: percent
  • Evaluates the whole process, requiring the correctness of both procedure and response. Scored as a percentage.
• SR — range: percent
  • Successful rate of the entire instance (thought, action, and arguments combined). Scored as a percentage.

Input / output format

Input: User instruction/query along with relevant tool documentation/API specifications.

Output: A structured sequence of tool calls (actions with parameters) followed by a final natural language response.

Scoring recipe

def evaluate(model_output, tool_docs, gold_answer):
    prompt = f'Tool docs: {tool_docs}\nGold answer: {gold_answer}\nModel output: {model_output}\nEvaluate Procedure, Response, Overall.'
    gpt4_response = call_gpt4(prompt)
    return parse_gpt4_scores(gpt4_response)

Common pitfalls

• Evaluation relies heavily on GPT-4 as an automated judge, which may introduce bias or inconsistency compared to human annotators.
• The simulated evaluation subset uses synthetically generated tools, so performance may not fully transfer to real-world API constraints or error handling.
• Success rate (SR) metrics on the GPT4Tools test set explicitly exclude non-tool-use cases, which could artificially inflate generalization scores.

Evidence (verbatim from paper)

To evaluate the models, we utilize GPT-4 for machine evaluation across all experiments, with an additional manual evaluation conducted specifically for the simulated subset. We prompt GPT-4 with the tool documentation and the standard answer from the human annotator and expect it to evaluate the performance in the following aspects:

• •

  Procedure: This metric evaluates the model’s proficiency in accurately selecting suitable actions, utilizing correct parameters, and avoiding redundant actions.

• •

  Response: This criterion measures whether the final response can satisfy the user’s instruction.

• •

  Overall: This metric evaluates the whole process, requiring the correctness of procedure and response.

Citation

@misc{tang2023toolalpaca,
  title={ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases},
  author={Tang et al. (2023)},
  year={2023},
  note={arXiv:2306.05301}
}

• arXiv: 2306.05301