Multivox Eval

multivox-eval

MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions — Selvakumar et al. (2025) (arXiv:2507.10859, 2025)

What this evaluates

Evaluates voice assistants' ability to jointly ground visual and paralinguistic speech cues (e.g., pitch, emotion, volume, background sounds) in context-aware responses. It specifically tests robustness against confounding samples that flip speech properties to prevent overreliance on unimodal priors.

Datasets

• MultiVox — total 1000; splits: test (-1)

Metrics

• visual grounding and non-verbal speech signals (primary) — range: other
  • Not explicitly defined in the provided text. The benchmark evaluates performance on jointly grounding visual and paralinguistic speech cues in context-aware responses.

Input / output format

Input: Speech-video or speech-image pairs paired with human-annotated dialogue turns.

Output: Context-aware responses from the evaluated voice assistant.

Scoring recipe

# Scoring recipe not specified in the provided text.
# Based on the benchmark description, evaluation likely involves:
# 1. Feed speech-video or speech-image pairs to the OVA.
# 2. Collect the assistant's context-aware response.
# 3. Assess response quality on visual grounding and non-verbal speech signal interpretation.
# 4. Compare against human-annotated dialogues or rubric-based scoring.
# Exact metrics and formulas are not included in this excerpt.

Common pitfalls

• Models may overrely on unimodal visual priors if confounding speech samples are not properly handled.
• Persistent failure in interpreting non-verbal speech signals (pitch, emotion, volume, background sounds) despite advances in multimodal reasoning.

Evidence (verbatim from paper)

MultiVox introduces the first human-spoken, multimodal benchmark for omni-modal voice assistants (OVAs), evaluating their ability to jointly ground visual and paralinguistic speech cues (e.g., pitch, emotion, volume, background sounds) in context-aware responses. It features 1,000 human-annotated dialogues with speech-video or speech-image pairs, including confounding samples that flip speech properties to prevent model overreliance on unimodal priors. Evaluation of 9 state-of-the-art OLMs reveals strong visual grounding but persistent failure in interpreting non-verbal speech signals, exposing a critical gap in speech understanding despite advances in multimodal reasoning.

Citation

@misc{selvakumar2025multivox,
  title={MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions},
  author={Selvakumar et al. (2025)},
  year={2025},
  note={arXiv:2507.10859}
}

• arXiv: 2507.10859