Express Emotion Recognition Eval

express-emotion-recognition-eval

Fluent but Unfeeling: The Emotional Blind Spots of Language Models — Shu et al. (2025) (arXiv:2509.09593, 2025)

What this evaluates

Evaluates language models' ability to recognize and decompose fine-grained human emotions from self-disclosed narratives. It probes whether models can align with human emotional expressions across 10 Plutchik-based dimensions (8 basic emotions + 2 sentiments) rather than just predicting surface-level emotion words.

Datasets

• EXPRESS — total 33679; splits: test (-1); repo https://github.com/Computing-for-Social-Good-CSG/express-emotion-recognition.git

Metrics

• Acc_L — range: [0, 1]
  • Lexical accuracy: the proportion of instances where the model's predicted emotion word exactly matches the human self-disclosed emotion word.
• Acc_V — range: [0, 1]
  • Vector accuracy: the proportion of instances where the model's predicted 10-dimensional emotion vector exactly matches the ground-truth human vector.
• F1_V (primary) — range: [0, 1]
  • Average Vector F-1 Score: macro-averaged F1 score computed across the 10 emotion dimensions of the predicted vs. ground-truth emotion vectors.

Input / output format

Input: Text of a human self-disclosed emotional experience (Reddit post), optionally accompanied by few-shot examples or CoT instructions.

Output: A predicted emotion word and/or a 10-dimensional binary/continuous vector representing the decomposed emotion according to Plutchik’s Wheel.

Scoring recipe

def score(predictions, golds):
    acc_l = sum(1 for p, g in zip(predictions, golds) if p.word == g.word) / len(predictions)
    acc_v = sum(1 for p, g in zip(predictions, golds) if p.vector == g.vector) / len(predictions)
    f1_v = 0.0
    for dim in range(10):
        tp = sum(1 for p, g in zip(predictions, golds) if p.vector[dim] == 1 and g.vector[dim] == 1)
        fp = sum(1 for p, g in zip(predictions, golds) if p.vector[dim] == 1 and g.vector[dim] == 0)
        fn = sum(1 for p, g in zip(predictions, golds) if p.vector[dim] == 0 and g.vector[dim] == 1)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1_v += 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    f1_v /= 10
    return acc_l, acc_v, f1_v

Common pitfalls

• CoT prompting consistently degrades performance on this task, contrary to its success in many reasoning benchmarks.
• Segmenting long posts into 512-token chunks does not disadvantage models with larger context windows, as full-post evaluation yields nearly identical results.
• Models often predict semantically plausible but contextually shallow emotions, overusing words like 'anxious' or 'frustrated' while missing human nuances like 'disheartened' or 'panicked'.

Evidence (verbatim from paper)

Here, we present our findings on the emotion recognition capabilities of LLMs evaluated on the EXPRESS dataset. ... $Acc_{L}$ ranged from 0.051 to 0.318, while $Acc_{V}$, slightly higher, ranged from 0.097 to 0.388. $F1_{V}$ ranged from 0.434 to 0.711, compared to a baseline of randomly generated vectors at 0.322.

Citation

@misc{shu2025fluent,
  title={Fluent but Unfeeling: The Emotional Blind Spots of Language Models},
  author={Shu et al. (2025)},
  year={2025},
  note={arXiv:2509.09593}
}

• arXiv: 2509.09593