Uro Bench Eval

uro-bench-eval

URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models — Yan et al. (2025) (arXiv:2502.17810, 2025)

What this evaluates

Evaluates end-to-end speech-to-speech dialogue models across three core dimensions: understanding, reasoning, and oral conversation. It probes multilingual proficiency, multi-turn dialogue handling, and the ability to generate paralinguistic and emotional cues in audio responses.

Datasets

• URO-Bench — total 40; splits: basic (-1), pro (-1)

Metrics

• Task Accomplish Score (primary) — range: percent
  • A composite metric measuring task completion and content quality across understanding, reasoning, and oral conversation dimensions.
• UTMOS — range: [0, 5]
  • Automatic mean opinion score for predicting natural speech quality.
• Matching Rate — range: percent
  • Percentage of pairwise model preferences where automated (GPT or rule-based) scores align with human evaluations.

Input / output format

Input: Audio prompts containing questions, instructions, or conversational turns in spoken dialogue.

Output: Audio responses generated by the end-to-end spoken dialogue model.

Scoring recipe

# 1. Task Accomplish Score & Content Quality
task_score = evaluate_task_completion(predictions, gold)
# 2. UTMOS for speech quality
utmos_val = utmos_model.predict(predictions.audio)
# 3. Matching Rate (Human vs AI consistency)
human_prefs = get_human_pairwise_prefs(config)
ai_scores = gpt_or_rule_based_score(predictions, gold)
matching_rate = sum(1 for h, a in zip(human_prefs, ai_scores) if h == a) / len(human_prefs)

Common pitfalls

• Confusing the basic and pro difficulty tracks, which test different levels of reasoning and audio understanding.
• Overlooking paralinguistic and emotional generation capabilities, which require specific rule-based or TTS-integrated scoring rather than pure text metrics.
• Assuming cascaded ASR+LLM+TTS pipelines are directly comparable to end-to-end SDMs without accounting for error propagation and speech-text alignment differences.

Evidence (verbatim from paper)

UTMOS of GPT-4o-Audio-Preview is lower than that of SLAM-Omni (Chen et al., 2024a) and GLM-4-Voice, but it achieves better speech-text alignment.

Citation

@misc{yan2025urobench,
  title={URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2502.17810}
}

• arXiv: 2502.17810