Voiceassistant Eval

voiceassistant-eval

VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing — Wang et al. (2025) (arXiv:2509.22651, 2025)

What this evaluates

Evaluates AI voice assistants across listening, speaking, and viewing capabilities. It probes audio understanding, multi-turn dialogue generation, role-play imitation, and multimodal vision-audio integration, measuring both content accuracy and speech naturalness.

Datasets

• VoiceAssistant-Eval — total 10497; splits: test (-1)

Metrics

• Final Task Score (primary) — range: percent
  • Product of three normalized sub-scores (Content Quality, Speech Quality via UTMOS, and Consistency via Modified WER), converted to a percentage. For role-play, Speaker Similarity is also factored in.

Input / output format

Input: Audio prompts (speech, music, or environmental sounds), optionally paired with images/video, along with task instructions or implicit context.

Output: Generated speech audio and a corresponding text response.

Scoring recipe

def compute_final_score(content_score, utmos_score, text_gen, text_ref):
    n = len([c for c in text_gen.lower() if c != ' '])
    m = len([c for c in text_ref.lower() if c != ' '])
    if min(n, m) < 10 and max(n, m) > 10:
        consistency = 1.0
    elif min(n, m) < 10 and max(n, m) <= 10:
        consistency = 0.0
    else:
        consistency = levenshtein(text_gen, text_ref) / max(n, m)
    final_score = content_score * utmos_score * consistency
    return final_score * 100

Common pitfalls

• Traditional WER is misleading for multiple-choice answers that only output a single letter; a length threshold is required.
• Models often show a capability mismatch, excelling at speaking tasks while struggling with listening/audio understanding.
• Proprietary models do not universally outperform open-source models across all 13 task categories.
• Role-play evaluation reveals a tension where higher content accuracy can correlate with lower speech naturalness.

Evidence (verbatim from paper)

To measure speech quality, we employ UTMOS, which provides an overall score reflecting fluency and naturalness. For evaluating consistency between text and speech modalities, we first transcribe the speech outputs using Whisper-Large-v3, and then compute a modified Word Error Rate (WER) between the transcriptions and the model’s text responses. ... The final score for each task is calculated by multiplying the three individual scores and then converting the result to a percentage.

Citation

@misc{wang2025voiceassistanteval,
  title={VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.22651}
}

• arXiv: 2509.22651