qhjqhj00 toolqa-eval
ToolQA: A Dataset for LLM Question Answering with External Tools — Zhuang et al. (2023) (arXiv:2306.13304, 2023)
What this evaluates
Evaluates whether LLMs can correctly answer questions that require interacting with external tools, rather than relying on pre-trained knowledge. It probes tool selection, multi-step tool chaining, and reasoning over execution traces in an open-ended setting.
Datasets
- ToolQA — total ?; splits: test (-1); repo https://github.com/night-chen/ToolQA
Metrics
success rate(primary) — range: percent- Percentage of instances where the normalized model prediction exactly matches the normalized ground-truth answer.
Input / output format
Input: Question text requiring external tool interaction, plus 8 tool-level demonstrations and system prompts.
Output: Final answer string (for standard LLMs) or interleaved reasoning traces and tool calls (for tool-augmented models), normalized for comparison.
Scoring recipe
def compute_success_rate(predictions, gold_answers):
norm_preds = [normalize(p) for p in predictions]
norm_golds = [normalize(g) for g in gold_answers]
correct = sum(1 for p, g in zip(norm_preds, norm_golds) if p == g)
return (correct / len(predictions)) * 100
Common pitfalls
- Questions are strictly out-of-distribution relative to pre-training data, so models cannot rely on memorized facts.
- Evaluation is open-ended regarding tool chains; only the final answer matters, not the specific tools used.
- Normalization of answers is required before exact matching to handle formatting variations.
Evidence (verbatim from paper)
To assess the performance of methods on the ToolQA benchmark, we normalize both the ground-truth answers and the model predictions to ensure uniformity in format. Success rates are then computed based on the exact match between these normalized answers.
Citation
@misc{zhuang2023toolqa,
title={ToolQA: A Dataset for LLM Question Answering with External Tools},
author={Zhuang et al. (2023)},
year={2023},
note={arXiv:2306.13304}
}
- arXiv: 2306.13304