qhjqhj00 av-speakerbench-eval
See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models — Nguyen et al. (2025) (arXiv:2512.02231, 2025)
What this evaluates
This benchmark probes fine-grained audiovisual reasoning in multimodal large language models, specifically requiring them to jointly determine who is speaking, what is being said, and when events occur within real-world video clips. It evaluates cross-modal fusion, temporal grounding, and speaker-centric perception through multiple-choice questions validated by human experts.
Datasets
- AV-SpeakerBench — total ?; splits: test (-1)
Metrics
MCQ accuracy(primary) — range: percent- Percentage of correctly answered multiple-choice questions out of the total number of questions.
Input / output format
Input: Audio-video clips paired with multiple-choice questions requiring speaker identification, speech content recognition, temporal grounding, or audio attribute estimation.
Output: A single selected option from the provided multiple-choice answers.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Models may exploit visual-only cues like mouth movements or gestures to answer questions without actually using the audio modality.
- Temporal grounding and overlapping speech perception are frequent failure points, leading to misaligned reasoning even when content is correctly perceived.
- Frame sampling policies vary significantly across models (e.g., 1 fps vs. fixed 8 frames), which can drastically affect temporal resolution and performance comparability.
Evidence (verbatim from paper)
The full evaluation results are presented in Table 2; all results are reported in MCQ accuracy. Here, we summarize the main takeaways. Existing models remain far from human performance. Human evaluation reaches an overall accuracy of 93.74%, confirming that the questions are clear and naturally solvable through audiovisual reasoning.
Citation
@misc{nguyen2025seehearandunderstand,
title={See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models},
author={Nguyen et al. (2025)},
year={2025},
note={arXiv:2512.02231}
}
- arXiv: 2512.02231