# Tau Bench Eval

> Evaluates the task completion success rate of LLM agents performing long-horizon, tool-using agentic workflows across customer service and software engineering domains. Probes the agent's ability to execute environment-altering (mutating) actions safely and maintain goal alignment over extended trajectories. Use when the user wants to benchmark on $ au$-Bench Airline, $ au$-Bench Retail, $ au$-Bench-V Air, $ au$-Bench-V Ret, SWE-Bench Verified, or asks about evaluating this task. Reports score.

- Skill: `qhjqhj00/tau-bench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/tau-bench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/tau-bench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/tau-bench-eval

---


# tau-bench-eval

> SABER: Small Actions, Big Errors - Safeguarding Mutating Steps in LLM Agents — Cuadron et al. (2025) (arXiv:2512.07850, 2025)

## What this evaluates

Evaluates the task completion success rate of LLM agents performing long-horizon, tool-using agentic workflows across customer service and software engineering domains. Probes the agent's ability to execute environment-altering (mutating) actions safely and maintain goal alignment over extended trajectories.

## Datasets

- **$	au$-Bench Airline** — total ?; splits: test (-1)
- **$	au$-Bench Retail** — total ?; splits: test (-1)
- **$	au$-Bench-V Air** — total ?; splits: test (-1)
- **$	au$-Bench-V Ret** — total ?; splits: test (-1)
- **SWE-Bench Verified** — total ?; splits: test (-1)

## Metrics

- `score` **(primary)** — range: percent
  - Number of successfully completed trajectories divided by total evaluated trajectories, multiplied by 100.

## Input / output format

**Input**: Task description and environment state; agent interacts via tool-calling interface over multiple turns.

**Output**: Sequence of tool calls/actions and final response, capped at 30 turns per episode.

## Scoring recipe

```python
success_count = 0
for run in range(3):
    for task in dataset:
        trajectory = run_agent(task, max_turns=30)
        if trajectory.is_successful():
            success_count += 1
return (success_count / (len(dataset) * 3)) * 100
```

## Common pitfalls

- High variance in original $	au$-Bench requires averaging over multiple runs to get stable scores.
- Dataset noise in original benchmarks masks true model capacity, making verified versions necessary for fair comparison.
- Auxiliary model pairing significantly impacts results, so cross-model comparisons must control for main/auxiliary configurations.

## Evidence (verbatim from paper)

> To reduce the variance present in $	au$-Bench, we report the average score over three runs. All evaluations are limited to 30 turns.

## Citation

```bibtex
@misc{cuadron2025saber,
  title={SABER: Small Actions, Big Errors - Safeguarding Mutating Steps in LLM Agents},
  author={Cuadron et al. (2025)},
  year={2025},
  note={arXiv:2512.07850}
}
```

- arXiv: 2512.07850