Emotionqueen Eval

emotionqueen-eval

EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models — Chen et al. (2024) (arXiv:2409.13359, 2024)

What this evaluates

Evaluates large language models' emotional intelligence and empathy by testing their ability to recognize key events, mixed events, implicit emotions, and user intent from real-world emotional scenarios, and to generate appropriate empathetic responses.

Datasets

• EmotionQueen — total 10000; splits: test (10000)

Metrics

• PASS rate (primary) — range: percent
  • Percentage of instances where the model correctly identifies the core event, emotion, or intent. Calculated as (correct identifications / total instances) * 100.
• WIN rate — range: percent
  • Percentage of instances where the model's generated response is rated as the most appropriate or empathetic by human annotators. Calculated as (winning responses / total instances) * 100.
• Average — range: percent
  • Arithmetic mean of the PASS rate and WIN rate, used for final model ranking.

Input / output format

Input: A user statement describing a real-world emotional scenario across five domains (achievements, family/friends, health, economy, accidents).

Output: A text response generated by the LLM intended to acknowledge the scenario and provide empathetic support or guidance.

Scoring recipe

# Human evaluation protocol
annotators = sample(annotator_pool, 3)
scores = []
for instance in test_set:
    response = model.generate(instance.prompt)
    ratings = [a.score(response) for a in annotators]
    ira = krippendorff_alpha(ratings)
    if ira < 0.7:
        replace_instance_with_new_one()
        continue
    scores.append(ratings)
pass_rate = (sum(1 for s in scores if s[0] == 'correct') / len(scores)) * 100
win_rate = (sum(1 for s in scores if s[0] == 'winner') / len(scores)) * 100
average = (pass_rate + win_rate) / 2

Common pitfalls

• High recognition accuracy (PASS rate) does not guarantee high empathetic response quality (WIN rate); the two metrics show little correlation across models.
• Human evaluation relies on a specific annotator demographic (female psychology graduate students), which may limit generalizability despite the authors' rationale.
• Automatic scoring via GPT-4 is used as a proxy but must be validated against human ratings, as alignment can vary across tasks and domains.

Evidence (verbatim from paper)

We randomly select 1000 questions each task generated by each LLM and enroll three volunteers to manually evaluate generated responses with the same metrics as GPT4. Our annotators are selected from a pool of female graduate students in psychology... To ensure the reliability and confidence of human ratings, we also calculate Inter-rater agreement of Krippendorff’s Alpha (IRA) to ensure the confidence of human ratings. For the controversial ratings which have low agreements (less than 0.7), we discard this statement and introduce another one. We rank them based on the average of PASS rate and WIN rate...

Citation

@misc{chen2024emotionqueen,
  title={EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2409.13359}
}

• arXiv: 2409.13359