Ultraeval Audio Eval

ultraeval-audio-eval

UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models — Shi et al. (2026) (arXiv:2601.01373, 2026)

What this evaluates

Evaluates audio foundation models across understanding, generation, and codec capabilities. It probes semantic accuracy, timbre fidelity, acoustic quality, and multilingual speech comprehension using a unified taxonomy and standardized benchmarks.

Datasets

• SpeechCMMLU — total 3519; splits: test (3519); repo https://github.com/OpenBMB/UltraEval-Audio
• SpeechHSK — total 170; splits: test (170); repo https://github.com/OpenBMB/UltraEval-Audio
• LibriSpeech — total ?; splits: test (-1)
• AISHELL-1 — total ?; splits: test (-1)

Metrics

• WER (primary) — range: [0, 1]
  • Word Error Rate: minimum number of word edits (insertions, deletions, substitutions) required to change the predicted transcript to the reference transcript, normalized by reference length.
• ASR-WER — range: [0, 1]
  • Computed by transcribing generated or reconstructed audio with a high-performance ASR model (e.g., Whisper-large-v3) and calculating WER on the transcriptions.
• SIM — range: [-1, 1]
  • Speaker embedding cosine similarity: cosine distance between WavLM-large embeddings of original and reconstructed audio to measure timbre fidelity.
• UTMOS — range: [1, 5]
  • Objective metric predicting overall speech naturalness and perceptual comfort.

Input / output format

Input: Audio files (speech, music, or environmental sounds) or text prompts, depending on the task (understanding, generation, or codec).

Output: Text transcriptions, translations, or answers for understanding tasks; synthesized or reconstructed audio files for generation/codec tasks.

Scoring recipe

def compute_wer(pred_text, ref_text):
    edits = levenshtein_distance(pred_text.split(), ref_text.split())
    return edits / max(len(ref_text.split()), 1)

def compute_sim(orig_audio, recon_audio):
    emb_orig = wavlm_large.encode(orig_audio)
    emb_recon = wavlm_large.encode(recon_audio)
    return cosine_similarity(emb_orig, emb_recon)

def compute_utmos(audio):
    return utmos_model.predict(audio)

Common pitfalls

• Inconsistent use of ASR models for computing ASR-WER across different benchmarks can skew semantic accuracy comparisons.
• Codec evaluation metrics (WER, SIM, UTMOS) measure different dimensions; relying solely on WER ignores timbre and acoustic quality degradation.
• TTS synthesis artifacts in SpeechCMMLU may introduce pronunciation errors if CER filtering is not strictly applied during dataset construction.

Evidence (verbatim from paper)

For semantics, we measure how well reconstructed audio preserves the original content using WER. Specifically, the reconstructed audio is transcribed by high-performance ASR models and compared to the original transcript. We employ Whisper-large-v3 for English and Paraformer-zh for Chinese. For timbre fidelity, we extract audio embeddings using WavLM-large fine-tuned on speaker verification, and compute the cosine similarity between embeddings of the original and reconstructed audio.

Citation

@misc{shi2026ultraevalaudio,
  title={UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models},
  author={Shi et al. (2026)},
  year={2026},
  note={arXiv:2601.01373}
}

• arXiv: 2601.01373