qhjqhj00 video-analytics-eval
AVA: Towards Agentic Video Analytics with Vision Language Models — Yan et al. (2025) (arXiv:2505.00254, 2025)
What this evaluates
Evaluates long-video understanding and agentic retrieval capabilities by testing models on temporal grounding, summarization, reasoning, and event causality across ultra-long video streams.
Datasets
- LVBench — total 1549; splits: test (1549)
- VideoMME-Long — total 900; splits: test (900)
- Ava-100 — total 120; splits: test (120)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered multiple-choice questions out of the total number of questions.
Input / output format
Input: A long video (duration varies from ~4100s to >10 hours) paired with a multiple-choice question.
Output: A single selected option from the provided multiple-choice answers.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
return correct / len(gold) * 100
Common pitfalls
- Videos are extremely long (up to >10 hours), requiring specialized indexing or retrieval rather than naive frame sampling.
- Questions cover diverse task types (temporal grounding, reasoning, etc.), so overall accuracy may mask performance on specific capabilities.
- Ava-100 is a newly proposed benchmark with manually annotated questions and GPT-4o-generated distractors, so results may not be directly comparable to older benchmarks without careful protocol alignment.
Evidence (verbatim from paper)
The accuracy is evaluated by analyzing Ava’s responses to multiple-choice questions included in the benchmarks.
Citation
@misc{yan2025ava,
title={AVA: Towards Agentic Video Analytics with Vision Language Models},
author={Yan et al. (2025)},
year={2025},
note={arXiv:2505.00254}
}
- arXiv: 2505.00254