Omnivideobench Eval

omnivideobench-eval

OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs — Li et al. (2025) (arXiv:2510.10689, 2025)

What this evaluates

Evaluates multimodal large language models' ability to jointly reason across visual and audio modalities in long-duration videos. It probes capabilities like cross-modal alignment, temporal dependency modeling, and understanding of low-semantic acoustic cues such as music and ambient sounds.

Datasets

• OmniVideoBench — total 1000; splits: test (1000)

Metrics

• accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly answered questions out of the total number of questions. Calculated separately for multiple-choice questions (MCQ) and open-ended question-answering (QA).

Input / output format

Input: Long-duration videos (up to 30 minutes) with synchronized audio, accompanied by a natural language question. Inputs may be provided as raw video frames + audio, silent frames + ASR transcripts, or varying frame counts (e.g., 32, 64, 128, 256 frames).

Output: For MCQ: a single selected option letter/text. For open-ended QA: a free-form textual response.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Multiple-choice question (MCQ) formats significantly overstate model performance compared to open-ended QA due to lexical cues and guessing opportunities.
• ASR transcripts cannot replace raw audio for tasks requiring deep acoustic comprehension (e.g., music or ambient sound understanding).
• Frame sampling density heavily influences results; sparse sampling misses key temporal events, especially in videos longer than 5 minutes.

Evidence (verbatim from paper)

To investigate whether the multiple-choice question (MCQ) format overstates model performance, we additionally evaluated several representative models on open-ended question-answering (QA) tasks, where no predefined answer options are provided. In this setting, models must directly generate textual responses, eliminating both the possibility of random guessing and any lexical cues potentially present in candidate options. In Table[4], the accuracy of all models drops significantly compared to their performance on multiple-choice questions.

Citation

@misc{li2025omnivideobench,
  title={OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2510.10689}
}

• arXiv: 2510.10689