Emovoice Eval

emovoice-eval

EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting — Guanrou Yang et al. (2025) (arXiv:2504.12867, 2025)

What this evaluates

Evaluates an LLM-based text-to-speech model's ability to generate emotionally expressive speech guided by natural language prompts, measuring content accuracy, emotional fidelity, and audio naturalness.

Datasets

• EmoVoice-DB — total ?; splits: test (-1), val (-1), train (-1)
• Secap — total 600; splits: test (600)

Metrics

• WER (primary) — range: [0, 1] or percent
  • Word Error Rate computed by transcribing synthesized speech with Whisper large-v3 and comparing against ground-truth text.
• Emotion Similarity — range: [-1, 1]
  • Cosine similarity between emotion embeddings extracted by emotion2vec from synthesized speech and ground-truth speech.
• Recall Rate — range: [0, 1]
  • Average proportion of correctly classified audio samples per emotion category using emotion2vec. Excludes disgusted, fearful, and surprised categories due to low recognition accuracy.
• UTMOS — range: [1, 5]
  • Perceptual audio quality and naturalness score generated by the UTMOS model.

Input / output format

Input: Text prompt (freestyle emotion description) + reference audio (neutral speech of same speaker for timbre) + target text content.

Output: Audio waveform (generated via flow matching and HiFi-GAN vocoder from 50 Hz semantic tokens).

Scoring recipe

# Content Consistency
pred_text = whisper_transcribe(audio_pred)
wer = compute_wer(pred_text, text_gold)

# Emotional Expressiveness
emb_pred = emotion2vec(audio_pred)
emb_gold = emotion2vec(audio_gold)
emotion_similarity = cosine_similarity(emb_pred, emb_gold)

pred_emotion = emotion2vec_classify(audio_pred)
# Exclude disgusted, fearful, surprised
recall_rate = average(accuracy_per_category(pred_emotion, emotion_gold))

# Naturalness
utmos_score = utmos_model(audio_pred)

Common pitfalls

• The paper excludes disgusted, fearful, and surprised emotions from Recall Rate evaluation due to low recognition accuracy of emotion2vec, which may skew results toward easier categories.
• Both Emotion Similarity and Recall Rate rely on emotion2vec, meaning the evaluation is bounded by the classifier's own biases and training data limitations rather than direct human perception.
• Timbre control uses a 'different neutral-emotion speech sample of the same speaker' as a prompt, introducing variability in reference audio that isn't standardized across all test cases.

Evidence (verbatim from paper)

We evaluate the content consistency of synthesized speech using the Word Error Rate (WER), with transcription results obtained from the Whisper large-v3*(Radford et al., [2023])* model. To assess emotional expressiveness, we adopt two metrics: Emotion Similarity and Recall Rate. Emotion Similarity is computed by extracting emotion embeddings from the synthesized speech using emotion2vec, and calculating the cosine similarity between these embeddings and those from the ground-truth speech. For Recall Rate, we also leverage emotion2vec to perform emotion classification on the generated speech. Due to insufficient training data and consequently low recognition accuracy of emotion2vec for the categories disgusted, fearful, and surprised, these emotions are excluded from the evaluation. Recall is computed as the proportion of correctly classified audio samples within each emotion category, and the final Recall Rate is the average value across all selected categories. In addition, we evaluate the naturalness and perceptual audio quality of the synthetic speech using the UTMOS score*(Saeki et al., [2022])*.

Citation

@misc{yang2025emovoice,
  title={EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting},
  author={Guanrou Yang et al. (2025)},
  year={2025},
  note={arXiv:2504.12867}
}

• arXiv: 2504.12867