Voxeval Eval

voxeval-eval

VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models — Cui et al. (2025) (arXiv:2501.04962, 2025)

What this evaluates

VoxEval probes the knowledge understanding and mathematical reasoning capabilities of end-to-end spoken language models (SLMs). It specifically evaluates how well these models comprehend spoken questions and generate accurate spoken answers under diverse audio conditions, including different speakers, speaking styles, and audio qualities.

Datasets

• VoxEval — total ?; splits: test (-1); repo https://github.com/dreamtheater123/VoxEval

Metrics

• accuracy (primary) — range: [0, 1]
  • Transcribe the model's spoken response using Whisper-large-v3, then apply string matching to extract the final multiple-choice answer (A, B, C, or D). Accuracy is the fraction of correctly matched answers.

Input / output format

Input: Audio questions synthesized from MMLU text using OpenAI TTS, optionally prepended with 5-shot in-context audio examples. Audio is often truncated to the last 80 seconds to fit model constraints.

Output: Spoken audio responses containing the model's answer.

Scoring recipe

def compute_accuracy(audio_responses, gold_answers):
    transcriptions = [whisper_transcribe(audio) for audio in audio_responses]
    predicted_answers = [extract_string_match(t) for t in transcriptions]
    correct = sum(1 for p, g in zip(predicted_answers, gold_answers) if p == g)
    return correct / len(gold_answers)

Common pitfalls

• Using chain-of-modality (text conversion) instead of end-to-end speech-to-speech generation drastically changes performance and latency.
• Chain-of-Thought prompting reduces SLM performance compared to direct answering, contrary to text-only models.
• String matching fails on CoT outputs due to lengthy reasoning steps; requires LLM-based extraction for accurate scoring.

Evidence (verbatim from paper)

To assess the spoken responses provided by the SLMs, we utilize the OpenAI ASR model whisper-large-v3 Radford et al. ([2023]) to convert their answers into text. Afterward, we apply string matching to determine the final answer (e.g., A, B, C, or D) from the transcription and calculate the accuracy.

Citation

@misc{cui2025voxeval,
  title={VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models},
  author={Cui et al. (2025)},
  year={2025},
  note={arXiv:2501.04962}
}

• arXiv: 2501.04962