Tau Voice Eval

tau-voice-eval

$\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains — Soham Ray et al. (arXiv:2603.13686, 2026)

What this evaluates

This benchmark evaluates full-duplex voice agents on real-world conversational tasks across retail, airline, and telecom domains. It jointly measures task completion success and real-time interaction quality, including responsiveness, latency, interruption handling, and selectivity under varying acoustic conditions like noise, diverse accents, and natural turn-taking.

Datasets

• $\tau^2$-bench — total 278; splits: test (278); repo https://github.com/sierra-research/tau2-bench

Metrics

• pass@1 (primary) — range: [0, 1]
  • Proportion of tasks completed successfully on a single attempt. Success is determined by deterministically comparing the final environment state (e.g., database records) against a gold standard.
• Responsiveness — range: [0, 1]
  • Average of Response Rate ($R_R$, proportion of user turns receiving a response) and Yield Rate ($R_Y$, proportion of interruptions where the agent yields within 2 seconds).
• Latency — range: other
  • Average of Response Latency ($L_R$, time from user utterance end to agent response) and Yield Latency ($L_Y$, time to stop speaking after an interruption).
• Interrupt — range: percent
  • Agent Interruption Rate ($I_A$), the proportion of turns where the agent speaks before the user finishes. Values >100% indicate multiple interruptions per turn.
• Selectivity — range: [0, 1]
  • Average of correctly ignoring backchannels ($S_{BC}$), vocal tics ($S_{VT}$), and non-directed speech ($S_{ND}$).

Input / output format

Input: Audio input simulating user speech (generated via TTS from a user simulator LLM) under controlled acoustic conditions (noise, accents, interruptions, frame drops). Models receive identical voice-specific system prompts.

Output: Real-time audio/text agent responses in a full-duplex conversational loop, with a maximum conversation duration of 1200 seconds and a 200ms processing tick.

Scoring recipe

def compute_metrics(env_final_state, gold_state, user_turns, agent_turns, interruptions, backchannels, vocal_tics, non_directed_speech):
    task_success = int(env_final_state == gold_state)
    R_R = sum(1 for t in user_turns if t in agent_turns) / len(user_turns)
    R_Y = sum(1 for i in interruptions if agent_turns.yields_within(i, 2.0)) / len(interruptions)
    L_R = mean([t.end - t.start for t in zip(user_turns, agent_turns)])
    L_Y = mean([t.stop - t.interrupt_time for t in interruptions])
    I_A = sum(1 for t in agent_turns if t.starts_before_user_ends(t)) / len(agent_turns)
    S_BC = mean([correctly_ignores(t) for t in backchannels])
    S_VT = mean([correctly_ignores(t) for t in vocal_tics])
    S_ND = mean([correctly_ignores(t) for t in non_directed_speech])
    return {
        "pass@1": task_success,
        "Responsiveness": (R_R + R_Y) / 2,
        "Latency": (L_R + L_Y) / 2,
        "Interrupt": I_A,
        "Selectivity": (S_BC + S_VT + S_ND) / 3
    }

Common pitfalls

• Clean vs Realistic conditions are often conflated; Clean strictly uses American accents with zero noise/interruptions, while Realistic adds diverse accents, environmental/burst noise, channel degradation, and natural turn-taking behaviors.
• The interrupt metric ($I_A$) is a rate, not a probability; values exceeding 100% are valid and indicate multiple interruptions per single user turn.
• Reproducibility relies on fixed seeds for acoustic/simulator components, not the LLM itself; stochastic agent responses are expected and do not invalidate the benchmark's controlled design.

Evidence (verbatim from paper)

Task Completion: Following $ au^{2}$-bench, tasks are fully verifiable: success is deterministically evaluated by comparing the end state of the environment (e.g., database records) against a gold standard. We report pass@1—the proportion of tasks completed successfully on a single attempt. Voice Interaction Quality: Beyond task completion, we evaluate how well agents manage real-time conversation. Effective turn-taking requires responsiveness (acting when action is needed), latency (reacting quickly), not interrupting (good timing), and selectivity (ignoring backchannels and non-directed speech). We measure: Responsiveness: Response Rate ($R_{R}$, proportion of user turns receiving a response) and Yield Rate ($R_{Y}$, proportion of interruptions where agent yields within 2s). Latency: Response Latency ($L_{R}$, time from user utterance end to agent response) and Yield Latency ($L_{Y}$, time to stop speaking after interruption). Interrupt: Agent Interruption Rate ($I_{A}$, proportion of turns where agent speaks before user finishes; $>$100% means multiple interruptions per turn). Selectivity: Correctly ignoring backchannels ($S_{BC}$), vocal tics ($S_{VT}$), and non-directed speech

Citation

@misc{ray2026tauvoice,
  title={$\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains},
  author={Soham Ray et al.},
  year={2026},
  note={arXiv:2603.13686}
}

• arXiv: 2603.13686