# Videollama3 Eval

> Evaluates multimodal foundation models on image and video understanding across multiple dimensions, including document and chart text recognition, mathematical reasoning, multi-image comprehension, general knowledge QA, long-form video comprehension, and temporal reasoning. Use when the user wants to benchmark on ChartQA, DocVQA, MathVista, VideoMME, Charades-STA, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/videollama3-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/videollama3-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/videollama3-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/videollama3-eval

---


# videollama3-eval

> VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding — Zhang et al. (2025) (arXiv:2501.13106, 2025)

## What this evaluates

Evaluates multimodal foundation models on image and video understanding across multiple dimensions, including document and chart text recognition, mathematical reasoning, multi-image comprehension, general knowledge QA, long-form video comprehension, and temporal reasoning.

## Datasets

- **ChartQA** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: testmini (-1)
- **VideoMME** — total ?; splits: test (-1)
- **Charades-STA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions, calculated as (correct predictions / total samples) * 100. For multiple-choice questions, exact match of the option letter is used. For open-ended questions, exact or normalized string match is applied.
- `mIoU` — range: [0, 1]
  - Mean Intersection over Union between predicted and ground-truth temporal intervals. Predicted intervals are extracted via regular expression from the model output.

## Input / output format

**Input**: Single image or video sequence (up to 180 frames) paired with a text prompt. Prompts are standardized per task type: MCQ prompts include 'Answer with the option letter from the given choices directly.', short-answer prompts include 'Answer the question with a single word or phrase.', and temporal grounding prompts append 'Please output the start and end timestamps in seconds'.

**Output**: Text string containing either a single uppercase option letter, a short word/phrase, or two numeric timestamps in seconds.

## Scoring recipe

```python
def score(predictions, golds, task_type):
    if task_type == 'mcq':
        return sum(1 for p, g in zip(predictions, golds) if p.strip().upper() == g.strip().upper()) / len(golds)
    elif task_type == 'short_answer':
        return sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g)) / len(golds)
    elif task_type == 'temporal_grounding':
        intervals = [extract_timestamps(p) for p in predictions]
        return mean_iou(intervals, golds)
    return 0.0
```

## Common pitfalls

- Temperature is fixed at 0.0 and sampling is disabled; enabling sampling or changing temperature will invalidate the reported scores.
- Strict prompt templates are required for each task type; omitting the instruction to 'Answer with the option letter...' causes parsing failures for MCQ benchmarks.
- Video evaluation caps frames at 180 and visual tokens at 16K; using dynamic frame sampling or longer contexts deviates from the official protocol.

## Evidence (verbatim from paper)

> For temporal grounding evaluation, we add an extra prompt “Please output the start and end timestamps in seconds" after the question. The numbers in the model response are extracted by regular expression, and then treated as one or multiple time intervals. Based on this strategy, we finally report the mIoU bewteen the ground-truth intervals and the predicted intervals.

## Citation

```bibtex
@misc{zhang2025videollama3,
  title={VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2501.13106}
}
```

- arXiv: 2501.13106