Multi Bench Eval

multi-bench-eval

MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models — Deng et al. (2025) (arXiv:2511.00850, 2025)

What this evaluates

Evaluates the emotional intelligence (EI) capabilities of spoken dialogue models in multi-turn interactive settings. It probes basic emotion understanding, advanced emotion support, paralinguistic analysis, and style inference across both Chinese and English dialogues.

Datasets

• MULTI-Bench — total 1500; splits: test (1500)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correct predictions on multi-choice tasks (e.g., emotion recognition, style inference, paralinguistic emotion inference). Calculated as the number of correct predictions divided by the total number of instances.
• LLM-judge score — range: other
  • Score assigned by Gemini-2.5-Pro (acoustic focus) or DeepSeek (textual focus) to model responses in multi-turn dialogues, assessing emotional accuracy and contextual responsiveness on a continuous scale.

Input / output format

Input: Multi-turn dialogue context (up to 10 turns) containing user prompts and model history, provided as audio and/or text.

Output: Model-generated spoken or textual response to the user's latest prompt.

Scoring recipe

def compute_metrics(predictions, golds, dialogue_turns):
    # Multi-choice tasks
    accuracy = sum(1 for pred, gold in zip(predictions, golds) if pred == gold) / len(golds)
    
    # Interactive dialogue (LLM judges)
    scores = []
    for turn in dialogue_turns:
        gemini_score = judge_gemini.evaluate(turn.response, turn.context, task_type)
        deepseek_score = judge_deepseek.evaluate(turn.response, turn.context, task_type)
        scores.append((gemini_score + deepseek_score) / 2)
    return accuracy, mean(scores)

Common pitfalls

• Multi-choice evaluation tasks consistently yield low accuracy across all systems, with models typically identifying only one correct option per question.
• Models often struggle with maintaining conversational memory across multi-turn interactions, leading to degraded performance in later turns.
• Language mixing occurs frequently in English dialogue tasks for models primarily trained on Chinese data.

Evidence (verbatim from paper)

Step Audio 2 leads in emotion recognition with 70.80%. Furthermore, it performs well on reasoning tasks. For example, it reaches 56.14% accuracy in best response style inference compared with 55.29% for Kimi Audio, and 40.00% in paralinguistic emotion inference compared with 35.28% for Kimi Audio.

Citation

@misc{deng2025multibench,
  title={MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models},
  author={Deng et al. (2025)},
  year={2025},
  note={arXiv:2511.00850}
}

• arXiv: 2511.00850