Emoscene Eval

emoscene-eval

Emotion Entanglement and Bayesian Inference for Multi-Dimensional Emotion Understanding — Kotaprolu et al. (2026) (arXiv:2604.00819, 2026)

What this evaluates

Probes a model's ability to perform context-aware, multi-dimensional emotion understanding by predicting Plutchik’s 8 basic emotions from rich textual scenarios. It specifically tests zero-shot multi-label emotion prediction and evaluates whether models can capture emotional entanglement (co-occurrence) rather than treating emotion dimensions as independent.

Datasets

• EmoScene — total 4731; splits: test (4731)

Metrics

• Macro F1 (primary) — range: [0, 1]
  • Unweighted mean of the F1 scores across the 8 Plutchik emotion classes. F1 per class is calculated as 2 * (Precision * Recall) / (Precision + Recall).
• Precision — range: [0, 1]
  • Ratio of correctly predicted positive instances to the total predicted positives for each emotion dimension.
• Recall — range: [0, 1]
  • Ratio of correctly predicted positive instances to the total actual positives for each emotion dimension.

Input / output format

Input: Context-rich textual scenario descriptions.

Output: Logit scores corresponding to 'Yes' and 'No' for each of the 8 Plutchik basic emotions, optionally accompanied by a natural language explanation and a confidence score.

Scoring recipe

def compute_macro_f1(predictions, gold):
    f1_scores = []
    for i in range(8):
        tp = sum(1 for p, g in zip(predictions, gold) if p[i] == 1 and g[i] == 1)
        fp = sum(1 for p, g in zip(predictions, gold) if p[i] == 1 and g[i] == 0)
        fn = sum(1 for p, g in zip(predictions, gold) if p[i] == 0 and g[i] == 1)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Models often treat the 8 emotion dimensions as independent, ignoring known emotional co-occurrence patterns (entanglement).
• Zero-shot prompting without task-specific fine-tuning leads to significant drops in Macro F1, especially for smaller models.
• Provided confidence scores do not reliably correlate with actual prediction accuracy.

Evidence (verbatim from paper)

Existing LLMs perform poorly in zero-shot multi-label emotion prediction (best Macro F1: 0.501), revealing limitations in context-aware emotional reasoning.

Citation

@misc{kotaprolu2026emoscene,
  title={Emotion Entanglement and Bayesian Inference for Multi-Dimensional Emotion Understanding},
  author={Kotaprolu et al. (2026)},
  year={2026},
  note={arXiv:2604.00819}
}

• arXiv: 2604.00819