# Toolqa Eval

> Evaluates whether LLMs can correctly answer questions that require interacting with external tools, rather than relying on pre-trained knowledge. It probes tool selection, multi-step tool chaining, and reasoning over execution traces in an open-ended setting. Use when the user wants to benchmark on ToolQA, or asks about evaluating this task. Reports success rate.

- Skill: `qhjqhj00/toolqa-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/toolqa-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/toolqa-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/toolqa-eval

---


# toolqa-eval

> ToolQA: A Dataset for LLM Question Answering with External Tools — Zhuang et al. (2023) (arXiv:2306.13304, 2023)

## What this evaluates

Evaluates whether LLMs can correctly answer questions that require interacting with external tools, rather than relying on pre-trained knowledge. It probes tool selection, multi-step tool chaining, and reasoning over execution traces in an open-ended setting.

## Datasets

- **ToolQA** — total ?; splits: test (-1); repo https://github.com/night-chen/ToolQA

## Metrics

- `success rate` **(primary)** — range: percent
  - Percentage of instances where the normalized model prediction exactly matches the normalized ground-truth answer.

## Input / output format

**Input**: Question text requiring external tool interaction, plus 8 tool-level demonstrations and system prompts.

**Output**: Final answer string (for standard LLMs) or interleaved reasoning traces and tool calls (for tool-augmented models), normalized for comparison.

## Scoring recipe

```python
def compute_success_rate(predictions, gold_answers):
    norm_preds = [normalize(p) for p in predictions]
    norm_golds = [normalize(g) for g in gold_answers]
    correct = sum(1 for p, g in zip(norm_preds, norm_golds) if p == g)
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Questions are strictly out-of-distribution relative to pre-training data, so models cannot rely on memorized facts.
- Evaluation is open-ended regarding tool chains; only the final answer matters, not the specific tools used.
- Normalization of answers is required before exact matching to handle formatting variations.

## Evidence (verbatim from paper)

> To assess the performance of methods on the ToolQA benchmark, we normalize both the ground-truth answers and the model predictions to ensure uniformity in format. Success rates are then computed based on the exact match between these normalized answers.

## Citation

```bibtex
@misc{zhuang2023toolqa,
  title={ToolQA: A Dataset for LLM Question Answering with External Tools},
  author={Zhuang et al. (2023)},
  year={2023},
  note={arXiv:2306.13304}
}
```

- arXiv: 2306.13304