Vbenchcomp Eval

vbenchcomp-eval

Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding? — Bo Feng et al. (2025) (arXiv:2505.14321, 2025)

What this evaluates

Evaluates video language models by disentangling question types into LLM-Answerable, Semantic, Temporal, and Others. It isolates true temporal and spatial understanding from language priors and static visual cues by computing accuracy exclusively on the Semantic and Temporal subsets.

Datasets

• LongVideoBench — total 1337; splits: test (1337)
• Egoschema — total 500; splits: test (500)
• NextQA — total 4996; splits: test (4996)
• VideoMME — total 2700; splits: test (2700)
• MLVU — total 2174; splits: test (2174)
• LVBench — total 1549; splits: test (1549)
• PerceptionTest — total 19140; splits: test (19140)

Metrics

• VBenchComp score (primary) — range: [0, 1]
  • Accuracy computed exclusively on questions categorized as Semantic or Temporal. It is calculated as the number of correct predictions on these filtered questions divided by the total number of Semantic and Temporal questions.
• Overall accuracy — range: [0, 1]
  • Standard accuracy computed across all questions in the benchmark, regardless of their diagnostic category.

Input / output format

Input: A video clip (typically sampled to 64 or 128 frames) paired with a natural language question.

Output: A natural language answer or multiple-choice selection, which is subsequently categorized into one of four diagnostic types (LLM-Answerable, Semantic, Temporal, Others) and compared against a gold reference.

Scoring recipe

def compute_vbenchcomp_score(predictions, gold_answers, categories):
    valid_mask = [cat in ['Semantic', 'Temporal'] for cat in categories]
    valid_preds = [p for p, m in zip(predictions, valid_mask) if m]
    valid_gold = [g for g, m in zip(gold_answers, valid_mask) if m]
    correct = sum(1 for p, g in zip(valid_preds, valid_gold) if p == g)
    return correct / len(valid_gold) if valid_gold else 0.0

Common pitfalls

• Relying solely on the overall accuracy score masks weaknesses in temporal reasoning, as models can achieve high scores by exploiting language priors on LLM-Answerable questions.
• Semantic questions are often frame-shuffling invariant, meaning models can answer correctly using static visual cues without understanding video dynamics, inflating performance on spatial tasks.
• Inconsistent frame sampling rates across benchmarks (e.g., 64 vs 128 frames) can artificially skew model comparisons if not normalized.

Evidence (verbatim from paper)

Based on the above analysis, we retain only the Semantic and Temporal questions from each benchmark to compute a focused evaluation score, denoted as the VBenchComp score. The results across models are shown in Figure[5]. Despite removing nearly 50% of the original questions (as detailed in Table[2]), the model rankings remain highly consistent with those based on the original scores.

Citation

@misc{feng2025vbenchcomp,
  title={Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?},
  author={Bo Feng et al. (2025)},
  year={2025},
  note={arXiv:2505.14321}
}

• arXiv: 2505.14321