Vimul Bench Eval

vimul-bench-eval

A Culturally-diverse Multilingual Multimodal Video Benchmark & Model — Bhuiyan et al. (2025) (arXiv:2506.07032, 2025)

What this evaluates

Evaluates video language models on multilingual, culturally-diverse video understanding across 14 languages and 15 domains. It probes the models' ability to answer multiple-choice and open-ended questions about short, medium, and long videos, with a specific focus on low-resource languages and cultural reasoning.

Datasets

• ViMUL-Bench — total 8000; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted options for multiple-choice questions, calculated as (correct predictions / total questions) * 100.
• correctness — range: percent
  • LLM-as-a-judge score evaluating how closely the model's open-ended response matches the ground truth, assessed using Phi-4-14B.

Input / output format

Input: Video frames (e.g., 32 frames sampled from the full video) and a textual query/prompt in one of 14 languages.

Output: For multiple-choice: a single selected option. For open-ended: a free-form text response in the query language.

Scoring recipe

def score(predictions, golds, question_type):
    if question_type == 'mcq':
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return (correct / len(golds)) * 100
    else:  # open-ended
        # LLM-as-a-judge (Phi-4-14B) compares prediction to gold
        # Returns a correctness score (e.g., 0/1 or percentage)
        return llm_judge_score(predictions, golds)

Common pitfalls

• Using GPT-based models to judge open-ended responses introduces inconsistency due to version updates and high computational cost.
• Evaluating with single-frame image inputs instead of full video sequences fails to capture spatio-temporal dynamics and significantly underestimates model capability.
• Assuming performance on high-resource languages (e.g., English) generalizes to low-resource languages, where accuracy drops sharply.

Evidence (verbatim from paper)

For multiple-choice questions, we provide the visual context and textual query to the LMMs, instructing them to select the best option, which is then directly compared to the ground truth. Performance is measured using accuracy, following established multiple-choice VQA benchmarks Romero et al. ([2024]); Bang et al. ([2023]); Zhu et al. ([2016]). For open-ended questions, we use the open-source multilingual LLM, Phi-4-14B Abdin et al. ([2024]) as a judge, ensuring consistency and reproducibility, unlike GPT-based models Shen et al. ([2023]); Stureborg et al. ([2024]), which are costly and inconsistent due to version updates. Performance is evaluated using correctness criteria, which measure how closely the model’s output matches the ground truth (see Sec. [A] in suppl. material for further detail).

Citation

@misc{bhuiyan2025vimul,
  title={A Culturally-diverse Multilingual Multimodal Video Benchmark & Model},
  author={Bhuiyan et al. (2025)},
  year={2025},
  note={arXiv:2506.07032}
}

• arXiv: 2506.07032