Humdial Eibench Eval

humdial-eibench-eval

HumDial-EIBench: A Human-Recorded Multi-Turn Emotional Intelligence Benchmark for Audio Language Models — Wang et al. (2026) (arXiv:2604.11594, 2026)

What this evaluates

Evaluates the emotional intelligence of audio language models across multi-turn dialogues. It probes four core capabilities: tracking emotional trajectories over time, reasoning about implicit emotional causes, generating empathetic responses, and resolving conflicts between acoustic and textual emotional signals.

Datasets

• HumDial-EIBench — total ?; splits: test (-1); repo https://github.com/ASLP-lab/HumDial-EIBench

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly selected options in multiple-choice tasks (Tasks 1, 2, and 4). Calculated as (number of correct predictions / total samples) * 100.
• Empathy/Naturalness Score — range: [1, 5]
  • Average rating on a 1-5 scale for text empathy (judged by LLMs), acoustic empathy, and response naturalness (judged by humans) in open-ended generation tasks (Task 3).

Input / output format

Input: Multi-turn audio-text dialogues. For Tasks 1, 2, and 4, the model receives context audio/text and multiple-choice options. For Task 3, the model receives conversational prompts and must generate a response.

Output: For Tasks 1, 2, and 4: a single selected multiple-choice option. For Task 3: a generated audio/text response.

Scoring recipe

# Tasks 1, 2, 4 (Multiple Choice)
correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
accuracy = (correct / len(golds)) * 100

# Task 3 (Open-ended)
# LLM judges score text empathy (D1) on a scale (e.g., 1-5)
# Human annotators score acoustic empathy (D2) and naturalness (D3) on a scale (e.g., 1-5)
empathy_score = (llm_judge_score + human_acoustic_score + human_naturalness_score) / 3

Common pitfalls

• Text-dominance bias: Models heavily over-rely on literal text semantics when acoustic and textual signals contradict, causing severe accuracy drops on conflict samples.
• LLM-as-a-judge instability: Automated text empathy scoring shows high variance (up to 1.0 point difference between judges), making it unreliable without human calibration.
• Cross-lingual interference: Models may generate responses in the wrong language (e.g., Chinese to English prompts), artificially deflating empathy scores.

Evidence (verbatim from paper)

Table 3: Accuracy (%) on Tasks 1 and 2. Bold: best in column; underline: second best.

Citation

@misc{wang2026humdialeibench,
  title={HumDial-EIBench: A Human-Recorded Multi-Turn Emotional Intelligence Benchmark for Audio Language Models},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.11594}
}

• arXiv: 2604.11594