Speechjudge Eval

speechjudge-eval

SpeechJudge: Towards Human-Level Judgment for Speech Naturalness — Zhang et al. (2025) (arXiv:2511.07931, 2025)

What this evaluates

This benchmark evaluates how well automated metrics and large audio-language models can judge speech naturalness and detect deepfakes compared to human preferences. It probes the alignment of computational quality scores with human perceptual judgments across multiple languages and TTS models.

Datasets

• SpeechJudge-Eval — total ?; splits: test (-1)

Metrics

• WER — range: percent
  • Word Error Rate computed by aligning ASR transcripts with reference text. Lower is better.
• SIM — range: [-1, 1]
  • Cosine similarity between WavLM TDNN speaker embeddings of generated and prompt samples.
• FAD — range: other
  • Fréchet Audio Distance computed using VGGish feature distributions between generated and reference audios.
• DNSMOS — range: [0, 5]
  • Perceptual speech quality score predicted by the DNSMOS model.
• Predicted MOS — range: [0, 5]
  • Mean Opinion Score predicted by the AudioBox-Aesthetics toolkit (CE/CU/PC/PQ variants).
• AASIST — range: [0, 1]
  • Deepfake detection accuracy using a heterogeneity-aware spectral-temporal graph model.
• ADV — range: [0, 1]
  • Deepfake detection accuracy using a w2v-bert-2.0 based multi-task model.
• AudioLLM Pairwise Accuracy (primary) — range: [0, 1]
  • Accuracy of AudioLLMs in correctly identifying the more natural audio in a generated vs. reference pair, using a standardized prompt.

Input / output format

Input: Single audio samples for automated metrics (WER, FAD, DNSMOS, MOS predictors, deepfake detectors), or pairs of audio samples (generated vs. reference/prompt) for pairwise naturalness scoring by AudioLLMs.

Output: Metric-specific numerical scores or classifications (e.g., WER transcript, cosine similarity value, FAD score, MOS rating, deepfake probability, or pairwise naturalness preference).

Scoring recipe

def evaluate(predictions, references):
    metrics = {}
    metrics['WER'] = compute_wer(predictions, references, asr='whisper-large-v3' if lang=='en' else 'paraformer-zh')
    metrics['SIM'] = cosine_sim(wavlm_embed(predictions), wavlm_embed(references))
    metrics['FAD'] = frechet_dist(predictions, references, feat='vggish')
    metrics['DNSMOS'] = run_dns_mos_script(predictions)
    metrics['MOS'] = audiobox_predict_mos(predictions)
    metrics['AASIST'] = aasist_detect(predictions)
    metrics['ADV'] = adv_detect(predictions)
    metrics['AudioLLM_Acc'] = pairwise_naturalness_acc(predictions, references, llm_api, prompt=TABLE_1_PROMPT)
    return metrics

Common pitfalls

• Cross-lingual WER evaluation requires switching ASR backends (Whisper for English, Paraformer for Chinese/code-switching), which can introduce inconsistent error baselines across languages.
• AudioLLM pairwise scoring is highly sensitive to the exact prompt format (Table 1) and API model versions, leading to non-reproducible results if not strictly pinned.
• Automated metrics like FAD, DNSMOS, and MOS predictors measure different acoustic qualities and often disagree with human naturalness judgments, requiring careful interpretation rather than direct substitution.

Evidence (verbatim from paper)

During the evaluation on the SpeechJudge-Eval Benchmark of Table [2], we adopt the following protocol for each model: WER: We employ Whisper-large-v3 for English texts, and Paraformer-zh for Chinese and code-switching texts. SIM: We compute the cosine similarity between the WavLM TDNN speaker embeddings of generated samples and the prompt samples. FAD: We use the officially released checkpoint, VGGish, to obtain the FADs of audios. DNSMOS: We use the officially released script to calculate the DNSMOS of audios.

Citation

@misc{zhang2025speechjudge,
  title={SpeechJudge: Towards Human-Level Judgment for Speech Naturalness},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2511.07931}
}

• arXiv: 2511.07931