Eicap Bench Eval

eicap-bench-eval

EICAP: Deep Dive in Assessment and Enhancement of Large Language Models in Emotional Intelligence through Multi-Turn Conversations — Nazar et al. (2025) (arXiv:2508.06196, 2025)

What this evaluates

Evaluates large language models' emotional intelligence (EI) capabilities across a four-layer taxonomy: emotional tracking, cause inference, appraisal, and emotionally appropriate response generation. It probes fine-grained subcategories including cultural sensitivity, valence judgment, and uncertainty calibration using multi-turn conversational contexts.

Datasets

• EICap-Bench — total ?; splits: test (-1)

Metrics

• macro-average accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered multiple-choice questions across all EI subcategories, averaged across dimensions. Reported as overall accuracy and delta accuracy (ΔAcc in percentage points) relative to a baseline after fine-tuning.

Input / output format

Input: Multi-turn conversational context or dialogue prompt requiring emotional intelligence reasoning, presented as a multiple-choice question.

Output: Model selects the correct option from the provided choices (MCQ format).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Delta accuracy (ΔAcc) is reported with statistical significance testing (10,000-sample paired bootstrap with Benjamini-Hochberg FDR correction), so raw point differences without confidence intervals or significance markers should not be interpreted as reliable.
• The benchmark evaluates multi-turn dialogues, but the MCQ format abstracts away generation quality; accuracy only measures option selection, not the quality or appropriateness of generated emotional responses.
• Narrow-slice LoRA fine-tuning on general corpora (UltraChat) often degrades foundation-layer EI performance, indicating that domain-aligned, task-annotated data is required for stable improvements.

Evidence (verbatim from paper)

Figure[2] summarizes zero-shot performance of three open-source LLM families Gemma-9B, LLaMA-3-8B, and Qwen-2.5-7B in both Base and Instruct variants on EICap-Bench. The evaluation spans fine-grained subcategories across all EI layers defined in the EICap taxonomy. Results show that Qwen-2.5-7B consistently outperforms the other models in macro-average accuracy across multiple EI dimensions, particularly in emotional inference and uncertainty calibration.

Citation

@misc{nazar2025eicap,
  title={EICAP: Deep Dive in Assessment and Enhancement of Large Language Models in Emotional Intelligence through Multi-Turn Conversations},
  author={Nazar et al. (2025)},
  year={2025},
  note={arXiv:2508.06196}
}

• arXiv: 2508.06196