Eve Emotion Recognition Eval

eve-emotion-recognition-eval

Evaluating Vision-Language Models for Emotion Recognition — Bhattacharyya et al. (2025) (arXiv:2502.05660, 2025)

What this evaluates

Evaluates vision-language models' ability to recognize evoked emotions from images in a zero-shot setting. It probes their robustness to prompt perturbations and measures sentiment bias in predicting positive vs. negative emotions.

Datasets

• EvE — total ?; splits: test (-1); repo https://github.com/sreebhattacharyya/Eve_Benchmark

Metrics

• weighted F1 score (primary) — range: [0, 1]
  • Computed by string-matching the model's parsed single-word prediction against the ground truth class label, then calculating the weighted F1 score across all emotion classes.

Input / output format

Input: An image I and a prompt P describing emotion labels for k discrete emotion classes.

Output: A single-word emotion prediction.

Scoring recipe

predictions = [parse_single_word(model_output) for model_output in model_outputs]
tp, fp, fn = {cls: 0 for cls in classes}, {cls: 0 for cls in classes}, {cls: 0 for cls in classes}
for p, g in zip(predictions, gold_labels):
    if p == g: tp[g] += 1
    elif p != g: fp[p] += 1
    if p != g: fn[g] += 1
precisions = [tp[c]/(tp[c]+fp[c]) if (tp[c]+fp[c])>0 else 0 for c in classes]
recalls = [tp[c]/(tp[c]+fn[c]) if (tp[c]+fn[c])>0 else 0 for c in classes]
supports = [sum(1 for g in gold_labels if g==c) for c in classes]
f1s = [2*p*r/(p+r) if (p+r)>0 else 0 for p, r in zip(precisions, recalls)]
weighted_f1 = sum(f * s for f, s in zip(f1s, supports)) / sum(supports)

Common pitfalls

• String-matching is used for evaluation, so any paraphrased or multi-word model output is automatically counted as incorrect.
• Model performance is highly sensitive to prompt structure, particularly the order of emotion labels provided in the prompt.
• Open-vocabulary or persona-based prompting significantly degrades F1 scores compared to simple classification prompts.

Evidence (verbatim from paper)

The responses are parsed and string-matched with the ground truth class labels, and weighted F1 scores are calculated.

Citation

@misc{bhattacharyya2025eve,
  title={Evaluating Vision-Language Models for Emotion Recognition},
  author={Bhattacharyya et al. (2025)},
  year={2025},
  note={arXiv:2502.05660}
}

• arXiv: 2502.05660