Vocalbench Eval

vocalbench-eval

VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Models — Heyang Liu et al. (2025) (arXiv:2505.15727, 2025)

What this evaluates

This benchmark evaluates end-to-end speech interaction models across semantic understanding, acoustic quality, conversational fluency, and robustness to noisy or diverse inputs. It probes the model's ability to generate natural, emotionally expressive speech while accurately following instructions and maintaining safety alignment. The evaluation also measures computational efficiency and latency to assess real-time usability.

Datasets

• VocalBench — total 9400; splits: test (9400)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correct answers for knowledge and reasoning tasks.
• following_rate (FR) — range: percent
  • Proportion of responses that correctly follow explicit instructions.
• refusal_rate (RR) — range: percent
  • Proportion of responses that appropriately refuse unsafe or harmful prompts.
• llm_score — range: other
  • LLM-as-judge rating on a 1-5 scale for creativity and open-ended dialogue.
• emotional_empathy_rate (EER) — range: percent
  • Proportion of responses demonstrating appropriate empathy in both semantics and acoustic tone.
• preserve_rate (PR) — range: percent
  • Ratio of model scores under distracted, accented, or multilingual conditions relative to clean/standard/monolingual baselines.
• rtf — range: other
  • Real-time factor measuring generation speed relative to audio duration.
• fcl — range: other
  • First chunk latency in milliseconds, measuring time to initial audio output.
• utmos — range: other
  • Automatic MOS score for speech fluency and naturalness.
• wer — range: percent
  • Word Error Rate measuring alignment between generated speech and reference transcript.
• overall_score (primary) — range: percent
  • Weighted composite score aggregating semantic, acoustic, chat, safety, latency, and robustness metrics.

Input / output format

Input: Speech prompts (text/audio) containing questions, instructions, or dialogue turns, including clean, distracted, accented, code-switched, and multilingual variants.

Output: Generated audio responses containing spoken text, requiring adherence to instructions, safety constraints, emotional tone, and natural prosody.

Scoring recipe

def compute_metrics(predictions, golds, prompts, audio_files, transcripts):
    correct = sum(1 for p, g in zip(predictions, golds) if matches(p, g))
    accuracy = (correct / len(golds)) * 100
    llm_score = llm_judge.evaluate(predictions, prompts) # 1-5 scale
    clean_scores = evaluate(audio_files_clean)
    noisy_scores = evaluate(audio_files_noisy)
    pr = (noisy_scores / clean_scores) * 100
    rtf = generation_time / audio_duration
    fcl = time_to_first_chunk_ms
    utmos = acoustic_model.predict_quality(audio_files)
    wer = asr.word_error_rate(audio_files, transcripts)
    overall = weighted_sum([accuracy, reasoning, creativity, fluency, clarity, chat, if, empathy, safety, latency, robustness])
    return accuracy, llm_score, pr, rtf, fcl, utmos, wer, overall

Common pitfalls

• Models often output structured text instead of natural speech, failing to generate recognizable spoken output in dialogues like math reasoning.
• Strong semantic backbones can paradoxically lower acoustic quality (UTMOS) due to complex phrasing requiring precise prosody and pause placement.
• Performance degrades significantly on long-form outputs due to repeated phrases, omitted fragments, or premature truncation in speech generation.
• Latency metrics (RTF/FCL) are hardware-dependent (e.g., L20 vs A100), making cross-model comparisons sensitive to the evaluation setup.

Evidence (verbatim from paper)

For instances with explicit answers, we used objective accuracy, along with the following rate (FR) for instruction following, and the refusal rate (RR) for safety alignment. For open-ended questions, including creativity and single-round dialogue, we adopt the LLM evaluation score on a 1-5 scale. In the empathy set, we defined the emotional empathy rate (EER), representing the proportion of the model’s response that demonstrates appropriate empathy in both semantics and acoustic tone. In code-switching, dialect and robustness set, we reported the preserve rate (PR), as the proportion of scores the model achieves under distracted, accented or multilingual conditions relative to clean, stardard or monolingual conditions.

Citation

@misc{liu2025vocalbench,
  title={VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Models},
  author={Heyang Liu et al. (2025)},
  year={2025},
  note={arXiv:2505.15727}
}

• arXiv: 2505.15727