qhjqhj00 videollama3-eval
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding — Zhang et al. (2025) (arXiv:2501.13106, 2025)
What this evaluates
Evaluates multimodal foundation models on image and video understanding across multiple dimensions, including document and chart text recognition, mathematical reasoning, multi-image comprehension, general knowledge QA, long-form video comprehension, and temporal reasoning.
Datasets
- ChartQA — total ?; splits: test (-1)
- DocVQA — total ?; splits: test (-1)
- MathVista — total ?; splits: testmini (-1)
- VideoMME — total ?; splits: test (-1)
- Charades-STA — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions, calculated as (correct predictions / total samples) * 100. For multiple-choice questions, exact match of the option letter is used. For open-ended questions, exact or normalized string match is applied.
mIoU— range: [0, 1]- Mean Intersection over Union between predicted and ground-truth temporal intervals. Predicted intervals are extracted via regular expression from the model output.
Input / output format
Input: Single image or video sequence (up to 180 frames) paired with a text prompt. Prompts are standardized per task type: MCQ prompts include 'Answer with the option letter from the given choices directly.', short-answer prompts include 'Answer the question with a single word or phrase.', and temporal grounding prompts append 'Please output the start and end timestamps in seconds'.
Output: Text string containing either a single uppercase option letter, a short word/phrase, or two numeric timestamps in seconds.
Scoring recipe
def score(predictions, golds, task_type):
if task_type == 'mcq':
return sum(1 for p, g in zip(predictions, golds) if p.strip().upper() == g.strip().upper()) / len(golds)
elif task_type == 'short_answer':
return sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g)) / len(golds)
elif task_type == 'temporal_grounding':
intervals = [extract_timestamps(p) for p in predictions]
return mean_iou(intervals, golds)
return 0.0
Common pitfalls
- Temperature is fixed at 0.0 and sampling is disabled; enabling sampling or changing temperature will invalidate the reported scores.
- Strict prompt templates are required for each task type; omitting the instruction to 'Answer with the option letter...' causes parsing failures for MCQ benchmarks.
- Video evaluation caps frames at 180 and visual tokens at 16K; using dynamic frame sampling or longer contexts deviates from the official protocol.
Evidence (verbatim from paper)
For temporal grounding evaluation, we add an extra prompt “Please output the start and end timestamps in seconds" after the question. The numbers in the model response are extracted by regular expression, and then treated as one or multiple time intervals. Based on this strategy, we finally report the mIoU bewteen the ground-truth intervals and the predicted intervals.
Citation
@misc{zhang2025videollama3,
title={VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding},
author={Zhang et al. (2025)},
year={2025},
note={arXiv:2501.13106}
}
- arXiv: 2501.13106