Humdial Eval

humdial-eval

The ICASSP 2026 HumDial Challenge: Benchmarking Human-like Spoken Dialogue Systems in the LLM Era — Zhao et al. (2026) (arXiv:2601.05564, 2026)

What this evaluates

This benchmark evaluates human-like spoken dialogue systems across two core capabilities: emotional intelligence (multi-turn emotion tracking, causal reasoning, and empathetic response generation) and full-duplex interaction (natural turn-taking, interruption handling, and noise rejection during concurrent listening and speaking). It uses authentic real-world conversations to measure long-term emotional consistency and cognitive synchronization.

Datasets

• HumDial — total ?; splits: test (-1)

Metrics

• Interruption success rate (primary) — range: [0, 1]
  • Proportion of correctly executed interruptions divided by total interruption attempts in full-duplex interaction scenarios.
• Rejection (silence maintenance) — range: [0, 1]
  • Proportion of background noise or invalid turns correctly ignored by the system without generating a response.
• Latency — range: other
  • Average time delay between user turn completion and system response initiation.
• Emotional resonance score — range: [0, 5]
  • Human or automated rating of empathetic vocal and textual response quality, typically on a Likert scale.

Input / output format

Input: Multi-turn real-world spoken dialogue context (audio and/or text), including user utterances, system history, and concurrent audio streams for full-duplex tracks.

Output: System responses (text and/or audio) for Track I, and real-time interaction decisions/actions (e.g., interrupt, continue, reject/silence) for Track II.

Scoring recipe

def score_humdial(predictions, gold):
    interruption_correct = sum(1 for p, g in zip(predictions, gold) if p.action == 'interrupt' and g.action == 'interrupt')
    interruption_rate = interruption_correct / max(len(predictions), 1)
    rejection_correct = sum(1 for p, g in zip(predictions, gold) if p.action == 'reject' and g.action == 'reject')
    rejection_rate = rejection_correct / max(len(predictions), 1)
    emotion_score = sum(p.rating for p in predictions) / len(predictions)
    return {
        'interruption_success_rate': interruption_rate,
        'rejection_rate': rejection_rate,
        'emotional_resonance': emotion_score
    }

Common pitfalls

• Distinguishing valid user turns from background noise remains difficult, leading to high false rejection rates.
• Models often excel at analyzing emotional logic but fail to generate empathetic vocal/textual responses that match the tracked trajectory.
• Optimizing for low latency in full-duplex mode frequently degrades noise rejection and interruption accuracy.

Evidence (verbatim from paper)

In Track II, top systems exhibited diverse strengths in real-time interaction (Table[3]). While Badcat achieved the highest Interruption success rate, Cookie_asr secured the top rank by delivering the best trade-off between low latency and robust noise rejection. However, scores for Rejection (silence maintenance) were consistently lower than Interruption, indicating that distinguishing valid user turns from background noise remains the primary hurdle for full-duplex systems.

Citation

@misc{zhao2026humdial,
  title={The ICASSP 2026 HumDial Challenge: Benchmarking Human-like Spoken Dialogue Systems in the LLM Era},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2601.05564}
}

• arXiv: 2601.05564