Widspeech Bench Eval

widspeech-bench-eval

WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wild — Linhao Zhang et al. (2025) (arXiv:2506.21875, 2025)

What this evaluates

Evaluates end-to-end speech-to-speech (S2S) language models on real-world conversational tasks, probing their ability to handle diverse query types, paralinguistic features (prosody, disfluencies), and robustness to background noise.

Datasets

• WildSpeech-Bench — total ?; splits: test (-1); repo https://github.com/Tencent/WildSpeech-Bench

Metrics

• Score (primary) — range: other
  • Mean score across five main categories: Text Creation (TC), Information Inquiry (II), Solution Request (SR), Opinion Exchange (OE), and Paralinguistic-Featured (PF) queries. PF is further averaged across five subcategories: Pause, Stress, Tone, Stutter, and near-Homophone. Scores are derived from query-aware checklists and human expert evaluation conducted directly on generated audio.

Input / output format

Input: Audio query containing conversational prompts, potentially augmented with background noise, human voice interference, or specific paralinguistic cues (e.g., pauses, stress, tone, stuttering, near-homophones).

Output: Generated audio response.

Scoring recipe

def compute_score(predictions, gold):
    categories = ['TC', 'II', 'SR', 'OE', 'PF']
    pf_subs = ['Pause', 'Stress', 'Tone', 'Stutter', 'Hphon.']
    cat_scores = {}
    for cat in categories:
        if cat == 'PF':
            cat_scores[cat] = mean([score_for_sub(sub) for sub in pf_subs])
        else:
            cat_scores[cat] = score_for_cat(cat)
    return mean(list(cat_scores.values()))
# Scores per instance are computed via query-aware checklists and validated against human expert ratings on generated audio.

Common pitfalls

• Strong speech-to-text (S2T) performance does not guarantee strong speech-to-speech (S2S) performance due to multi-modal generation complexities.
• Naive ASR-LLM-TTS pipelines lose crucial paralinguistic information during transcription, causing significant drops on stress and tone tasks.
• High performance in clean audio environments does not transfer to real-world noisy scenarios, particularly with human voice background noise.

Evidence (verbatim from paper)

To validate our framework, we benchmarked automated scores against rigorous human expert evaluations. Crucially, these evaluations were conducted directly on the generated audio, not text transcriptions, to capture essential qualities like prosody, tone, and clarity (see Appendix [B] for the protocol). As shown in Figure [4], WildSpeech achieves a significantly higher Pearson correlation with human scores than the VoiceBench baseline.

Citation

@misc{zhang2025widspeechbench,
  title={WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wild},
  author={Linhao Zhang et al. (2025)},
  year={2025},
  note={arXiv:2506.21875}
}

• arXiv: 2506.21875