Mme Emotion Eval

mme-emotion-eval

MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models — Fan Zhang et al. (2025) (arXiv:2508.09210, 2025)

What this evaluates

This benchmark evaluates the emotional intelligence of multimodal large language models (MLLMs) by testing their ability to recognize emotions, perform causal reasoning about emotional triggers, and generate structured chain-of-thought explanations. It probes fine-grained sentiment analysis, multimodal fusion capabilities, and reasoning depth across diverse video scenarios.

Datasets

• MME-Emotion — total 6500; splits: test (6500)

Metrics

• CoT-S (primary) — range: percent
  • Chain-of-Thought Score: Measures the quality of the model's step-by-step reasoning for emotional triggers. Evaluated via a multi-agent framework combining automated parsing and human validation.
• Rec-S — range: percent
  • Recognition Score: Exact-match accuracy for direct emotion classification from the input video clip.
• Rea-S — range: percent
  • Reasoning Score: Accuracy of the model's causal explanation for why an emotion is present, validated via multi-agent/human assessment.
• Avg Step — range: other
  • Average Step Count: Mean number of reasoning steps generated per response.
• Avg Token — range: other
  • Average Token Count: Mean number of tokens generated per response.

Input / output format

Input: Video clips containing audio and visual modalities, accompanied by zero-shot text prompts. Prompts are adapted per model capability (e.g., replacing 'video' with 'audio' or 'video frames' as needed).

Output: Text responses containing direct emotion labels, reasoning steps, and/or chain-of-thought explanations.

Scoring recipe

def compute_metrics(predictions, gold):
    # Recognition Score (Exact Match)
    rec_score = 1.0 if predictions['label'] == gold['label'] else 0.0
    
    # Reasoning & CoT Scores (Multi-agent/Human Validation)
    rea_score = judge_agent.validate_reasoning(predictions['reasoning'], gold['reasoning'])
    cot_score = judge_agent.validate_cot_quality(predictions['cot'], gold['cot'])
    
    # Length Metrics
    avg_step = count_steps(predictions['full_response'])
    avg_token = count_tokens(predictions['full_response'])
    
    return {'Rec-S': rec_score, 'Rea-S': rea_score, 'CoT-S': cot_score, 'Avg Step': avg_step, 'Avg Token': avg_token}

Common pitfalls

• Multimodal inputs often contain redundant or conflicting emotional cues, causing omnimodal models to underperform simpler unimodal or vision-only models.
• Models trained primarily on in-the-wild data struggle to generalize to controlled in-lab scenarios, leading to significant performance drops on lab-based tasks.
• Direct recognition scores are often decoupled from reasoning quality, meaning high Rec-S does not guarantee accurate or coherent CoT generation.

Evidence (verbatim from paper)

We showcase the overall performance comparison on MME-Emotion, covering three performance metrics: recognition score (Rec-S), reasoning score (Rea-S), and Chain-of-Thought score (CoT-S) as well as two response length metrics: average step count (Avg Step) and average token count (Avg Token), as shown in Table [2].

Citation

@misc{zhang2025mmeemotion,
  title={MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models},
  author={Fan Zhang et al. (2025)},
  year={2025},
  note={arXiv:2508.09210}
}

• arXiv: 2508.09210