Tau2 Bench Eval

tau2-bench-eval

$\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment — Barres et al. (2025) (arXiv:2506.07982, 2025)

What this evaluates

Evaluates conversational agents' ability to collaborate with a user simulator in a dual-control environment where both parties share tool access to a dynamic world. It probes coordination, communication under decentralized control, and adherence to domain-specific policies while resolving multi-step tasks.

Datasets

• $\tau^2$-Bench — total 200; splits: test (200); repo https://github.com/sierra-research/tau2-bench

Metrics

• pass^1 (primary) — range: [0, 1]
  • The fraction of tasks for which the agent achieves successful completion in a single deterministic run. Calculated as (number of tasks with successful trajectory) / (total number of tasks).

Input / output format

Input: Domain policy document, task-specific instructions, OpenAI-format tool definitions, and conversation history. The agent receives generic guidelines plus domain policies; the user simulator receives generic guidelines plus task-specific instructions.

Output: Conversational text responses and structured function/tool calls in OpenAI format.

Scoring recipe

successes = 0
for task in dataset:
    trajectory = agent.run(task, tools, policy, user_sim)
    if environment.verify_success(trajectory):
        successes += 1
return successes / len(dataset)

Common pitfalls

• LLM temperature must be strictly set to 0 to ensure deterministic outputs; non-zero temperatures invalidate pass^k comparisons.
• Ablation settings (No-User, Oracle Plan) alter tool access and information availability, so their scores cannot be directly compared to the default dual-control setting.
• User simulator errors (critical vs. benign) can confound agent performance; critical errors that prevent task completion must be distinguished from benign ones to avoid unfairly penalizing the agent.

Evidence (verbatim from paper)

We computed performance metrics on the verified $ au^2$-bench domains (retail and airline) and on our new telecom domain (see Figure 3). Our findings indicate that the telecom domain presents a greater challenge, exhibiting an overall lower success rate compared to other domains. gpt-4.1 pass^1 drops from 74% / 56% for retail and airline respectively to 34% for telecom.

Citation

@misc{barres2025tau2bench,
  title={$\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment},
  author={Barres et al. (2025)},
  year={2025},
  note={arXiv:2506.07982}
}

• arXiv: 2506.07982