Mmeverse Bench Eval

mmeverse-bench-eval

Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding — Peng et al. (2026) (arXiv:2601.16449, 2026)

What this evaluates

This benchmark evaluates multimodal large language models on emotion recognition and emotion reasoning across diverse video clips. It probes the model's ability to extract and fuse audio, visual, and textual cues to predict categorical emotion labels and generate structured, modality-grounded explanations.

Datasets

• MMEVerse-Bench — total 166000; splits: train (130000), test (36000); repo https://github.com/ooochen-30/Emotion-LLaMA-v2
• EMER — total ?; splits: test (-1)

Metrics

• Avg-18 (primary) — range: percent
  • Average performance across all 18 test sets on MMEVerse-Bench. Identical to Avg-9 for MER-UniBench datasets, but includes additional unique datasets.
• hit rate — range: [0, 1]
  • Percentage of correctly predicted emotion labels out of total instances. Used for MER2023, MER2024, MELD-e, and IEMOCAP.
• weighted average F-score (WAF) — range: [0, 1]
  • F1-score computed per class and averaged, weighted by class support. Used for MOSI, MOSEI, SIMS, and SIMS-v2.
• mean Average Precision (mAP) — range: [0, 1]
  • Average precision across all emotion classes for multi-label tasks. Used for MAFW-m and BOLD.
• Clue Overlap — range: [0, 10]
  • GPT-4o scored overlap between model-generated emotion-related clues and ground-truth triggers on a 0-10 scale.
• Label Overlap — range: [0, 10]
  • GPT-4o scored overlap between model-summarized emotional states and ground-truth labels on a 0-10 scale.

Input / output format

Input: Multimodal prompt template concatenating: (1) 64-token audio sequence from Whisper-large-v3, (2) temporal visual sequence from EVA-ViT-G (16 uniformly sampled frames, spatially pooled), (3) global visual context from middle frame [CLS] token, and (4) text instruction/query. All embeddings projected to 4096-dim space and concatenated.

Output: For recognition: categorical emotion label. For reasoning: structured natural language explanation detailing emotion triggers, summarized emotional states, and cross-modal reasoning steps.

Scoring recipe

def score_recognition(preds, golds, dataset_name):
    if dataset_name in ['MAFW-m', 'BOLD']:
        return compute_mAP(preds, golds)
    elif dataset_name in ['MOSI', 'MOSEI', 'SIMS', 'SIMS-v2']:
        return compute_WAF(preds, golds)
    elif dataset_name in ['MER2023', 'MER2024', 'MELD-e', 'IEMOCAP']:
        return accuracy_score(golds, preds)
    else:
        return compute_avg_F_score_across_wheels(preds, golds)

def score_reasoning(preds, golds):
    prompt = build_overlap_prompt(preds, golds)
    clue_score = gpt4o_score(prompt, dimension='clue_overlap')
    label_score = gpt4o_score(prompt, dimension='label_overlap')
    return clue_score, label_score

Common pitfalls

• Using different GPT model versions for EMER reasoning scoring breaks cross-model comparability; results are only valid when evaluated with a consistent GPT version.
• Confusing Avg-9 (identical to MER-UniBench performance) with Avg-18 (average across all 18 MMEVerse-Bench test sets).
• Applying a single metric across all datasets instead of using task-appropriate metrics (e.g., hit rate for MER2023/MELD, WAF for MOSI/MOSEI, mAP for multi-label MAFW-m/BOLD).

Evidence (verbatim from paper)

The evaluation protocol for MMEVerse-Bench ensures fair comparability by strictly following the metrics of MER-UniBench for all overlapping datasets, while applying standard task-appropriate metrics for datasets unique to our collection. Specifically, we use hit rate for MER2023, MER2024, MELD-e, and IEMOCAP; weighted average F-score (WAF) for MOSI, MOSEI, SIMS, and SIMS-v2; and average F-score across emotion wheels for OV-MERD+. For our unique datasets, we adopt mean Average Precision (mAP) for multi-label tasks (MAFW-m and BOLD) and standard accuracy for the remaining datasets.

Citation

@misc{peng2026emotionllmav2,
  title={Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding},
  author={Peng et al. (2026)},
  year={2026},
  note={arXiv:2601.16449}
}

• arXiv: 2601.16449