# Simpletoolhallubench Eval

> simpletoolhallubench-eval

- Skill: `qhjqhj00/simpletoolhallubench-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/simpletoolhallubench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/simpletoolhallubench-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/simpletoolhallubench-eval

---


# simpletoolhallubench-eval

> The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination — Chenlong Yin et al. (2025) (arXiv:2510.22977, 2025)

## What this evaluates

Probes an agent's ability to abstain from tool use when no appropriate tools are available, specifically measuring its tendency to hallucinate tool invocations in two controlled scenarios: when no tools are provided and when irrelevant distractor tools are present.

## Datasets

- **SimpleToolHalluBench** — total 296; splits: test (296)

## Metrics

- `R_NTA` **(primary)** — range: [0, 1]
  - Hallucination rate for the No-Tool-Available task, calculated as the fraction of responses flagged by an LLM-as-judge as hallucinating tool invocations: R_NTA = H_NTA / N_NTA.
- `R_DT` — range: [0, 1]
  - Hallucination rate for the Distractor-Tool task, calculated as the fraction of responses flagged by an LLM-as-judge as hallucinating tool invocations: R_DT = H_DT / N_DT.

## Input / output format

**Input**: User query paired with a system prompt that either provides no tools (NTA) or provides an irrelevant distractor tool (DT).

**Output**: Model's generated response, which may include a tool invocation, a refusal/abstention, or a hallucinated tool call.

## Scoring recipe

```python
# For each task (NTA or DT):
N = total number of samples in the task
H = 0
for response in model_responses:
    if llm_as_judge_flags_hallucination(response):
        H += 1
rate = H / N
return rate
```

## Common pitfalls

- Queries are explicitly designed to be impossible without the specific required tool, so models should not rely on internal knowledge to answer them.
- Scoring relies on an LLM-as-judge to flag hallucinations, which may introduce variability or bias compared to deterministic tool-existence checks.
- The benchmark separates 'NTA' (no tools provided) and 'DT' (irrelevant tool provided) scenarios, so results should not be averaged without noting the distinct failure modes.

## Evidence (verbatim from paper)

> We calculate the hallucination rate for each task independently. These rates are defined as the fraction of responses flagged by the LLM-as-judge: R_NTA = H_NTA / N_NTA and R_DT = H_DT / N_DT, where H_NTA and H_DT are the counts of hallucinated responses in their respective tasks, and N_NTA and N_DT are the total number of samples in each task set.

## Citation

```bibtex
@misc{yin2025reasoningtrap,
  title={The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination},
  author={Chenlong Yin et al. (2025)},
  year={2025},
  note={arXiv:2510.22977}
}
```

- arXiv: 2510.22977