Emotrans Eval

emotrans-eval

EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs — He Hu et al. (2026) (arXiv:2604.23348, 2026)

What this evaluates

Evaluates multimodal LLMs on understanding, reasoning, and predicting fine-grained emotion transitions in short video clips. It probes capabilities across four progressive tasks: detecting whether an emotion change occurs, identifying before/after emotion states, generating evidence-grounded reasoning for transitions, and predicting the next emotion state.

Datasets

• EmoTrans — total 3274; splits: test (3274); repo https://github.com/Emo-gml/EmoTrans

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly predicted labels for classification tasks (ECD, ESI, NEP). Calculated as (correct predictions / total instances) * 100.
• BERTScore — range: [0, 1]
  • Contextual embedding-based similarity between generated reasoning explanations and ground-truth rationales, computed using BERT embeddings.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap between generated and reference text, measuring structural similarity.
• BLEU4 — range: [0, 1]
  • 4-gram precision with brevity penalty comparing generated explanations to reference rationales.
• LLM-Score — range: other
  • Automated scoring of generated reasoning quality using a separate LLM judge against ground-truth explanations.

Input / output format

Input: Video clip (audio+visual) with aligned modalities, person identification attributes (e.g., clothing, position), and a task-specific natural language prompt.

Output: Task-dependent: (1) Binary label (0 or 1) for emotion change; (2) Tuple of before/after emotion labels (e_bef, e_aft) or single label e; (3) Natural language rationale explaining the transition or stability; (4) Single predicted emotion label.

Scoring recipe

def score_emotrans(predictions, golds, task):
    if task in ['ECD', 'ESI', 'NEP']:
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return correct / len(golds) * 100
    elif task == 'ETR':
        bert = compute_bertscore(predictions, golds)
        rouge = compute_rouge_l(predictions, golds)
        bleu = compute_bleu4(predictions, golds)
        llm = compute_llm_score(predictions, golds)
        return {'BERTScore': bert, 'ROUGE-L': rouge, 'BLEU4': bleu, 'LLM-Score': llm}

Common pitfalls

• Multi-person videos require per-person predictions; models must correctly attribute emotions to specific individuals using provided attributes, otherwise predictions are misaligned.
• Temporal tolerance is 0.5 seconds for change points; exact timestamp matching will incorrectly penalize otherwise valid predictions.
• Emotion labels are strictly categorical (7 classes: Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral); dimensional or intensity-based outputs are invalid.

Evidence (verbatim from paper)

For classification tasks (ECD, ESI, and NEP), results are reported in Accuracy (%). Avg. denotes the mean Accuracy over ECD, ESI, and NEP. Red and blue denote the best and second-best results among all models.

Citation

@misc{he2026emotrans,
  title={EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs},
  author={He Hu et al. (2026)},
  year={2026},
  note={arXiv:2604.23348}
}

• arXiv: 2604.23348