Tau Bench Eval

tau-bench-eval

SABER: Small Actions, Big Errors - Safeguarding Mutating Steps in LLM Agents — Cuadron et al. (2025) (arXiv:2512.07850, 2025)

What this evaluates

Evaluates the task completion success rate of LLM agents performing long-horizon, tool-using agentic workflows across customer service and software engineering domains. Probes the agent's ability to execute environment-altering (mutating) actions safely and maintain goal alignment over extended trajectories.

Datasets

• $ au$-Bench Airline — total ?; splits: test (-1)
• $ au$-Bench Retail — total ?; splits: test (-1)
• $ au$-Bench-V Air — total ?; splits: test (-1)
• $ au$-Bench-V Ret — total ?; splits: test (-1)
• SWE-Bench Verified — total ?; splits: test (-1)

Metrics

• score (primary) — range: percent
  • Number of successfully completed trajectories divided by total evaluated trajectories, multiplied by 100.

Input / output format

Input: Task description and environment state; agent interacts via tool-calling interface over multiple turns.

Output: Sequence of tool calls/actions and final response, capped at 30 turns per episode.

Scoring recipe

success_count = 0
for run in range(3):
    for task in dataset:
        trajectory = run_agent(task, max_turns=30)
        if trajectory.is_successful():
            success_count += 1
return (success_count / (len(dataset) * 3)) * 100

Common pitfalls

• High variance in original $ au$-Bench requires averaging over multiple runs to get stable scores.
• Dataset noise in original benchmarks masks true model capacity, making verified versions necessary for fair comparison.
• Auxiliary model pairing significantly impacts results, so cross-model comparisons must control for main/auxiliary configurations.

Evidence (verbatim from paper)

To reduce the variance present in $ au$-Bench, we report the average score over three runs. All evaluations are limited to 30 turns.

Citation

@misc{cuadron2025saber,
  title={SABER: Small Actions, Big Errors - Safeguarding Mutating Steps in LLM Agents},
  author={Cuadron et al. (2025)},
  year={2025},
  note={arXiv:2512.07850}
}

• arXiv: 2512.07850