Emoval Emotion Dialogue Eval

emoval-emotion-dialogue-eval

EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions — Kai Chen et al. (2024) (arXiv:2409.18042, 2024)

What this evaluates

Evaluates an omni-modal language model's ability to engage in end-to-end spoken dialogue with vivid emotional control. It probes the model's dialogue quality, text generation accuracy under different input modalities, and its capability to control and classify speech styles/emotions.

Datasets

• EMOVA-EmotionDialogue-Test — total ?; splits: test (-1)

Metrics

• end-to-end spoken dialogue score (primary) — range: [0, 10]
  • Average score from 0 to 10 assigned by GPT-4o to assess dialogue performance based on generated speech quality and emotional expression.
• unit-input-text-output score — range: [0, 10]
  • Score assessing the quality of textual responses when the model receives speech units as input, bypassing speech synthesis errors.
• text-input-text-output score — range: [0, 10]
  • Score assessing textual response quality when the model receives ground-truth user instruction texts as input.
• style label classification accuracy — range: [0, 1]
  • Percentage of correctly predicted style/emotion labels from the model's output compared to ground truth.
• style controllability — range: other
  • Measured using a confusion matrix comparing conditional style labels provided to the detokenizer against recognized style labels in the generated speech.

Input / output format

Input: Input image and user instructions provided as speech units (or ground-truth text for text-input variant).

Output: Text responses, style labels, and corresponding speech units.

Scoring recipe

def evaluate(predictions, gold):
    # LLM-as-judge for dialogue scores (0-10)
    scores = [gpt4o_judge(f"Rate dialogue 0-10: {inp}, {pred}") for inp, pred in zip(inputs, predictions)]
    avg_score = sum(scores) / len(scores)
    
    # Style classification accuracy
    correct = sum(1 for p, g in zip(predictions['style_labels'], gold['style_labels']) if p == g)
    accuracy = correct / len(gold['style_labels'])
    
    # Controllability via confusion matrix
    confusion = compute_confusion_matrix(predictions['generated_styles'], gold['recognized_styles'])
    return avg_score, accuracy, confusion

Common pitfalls

• The evaluation relies entirely on GPT-4o for automated scoring, which may introduce judge bias or inconsistency compared to human evaluation.
• ASR/TTS errors in the end-to-end pipeline can unfairly penalize the dialogue score; the protocol explicitly disentangles this by comparing unit-input vs text-input scores.
• The test set is split from the model's own synthesized instruction-tuning data, raising potential data leakage or distribution shift concerns.

Evidence (verbatim from paper)

To ensure comprehensive evaluation, we propose the following evaluation metrics: 1. End-to-end spoken dialogue score assesses the model’s dialogue performance based on the generated speeches, with a score ranging from 0 to 10, reporting the average. ... Due to the lack of emotionally rich spoken dialogue evaluation datasets, we split a test set from our synthesized omni-modal instruction-tuning data (Sec.[4.1]). GPT-4o is used for automated evaluation.

Citation

@misc{chen2024emova,
  title={EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions},
  author={Kai Chen et al. (2024)},
  year={2024},
  note={arXiv:2409.18042}
}

• arXiv: 2409.18042