H2vu Benchmark Eval

h2vu-benchmark-eval

H2VU-Benchmark: A Comprehensive Benchmark for Hierarchical Holistic Video Understanding — Qi Wu et al. (2025) (arXiv:2503.24008, 2025)

What this evaluates

Evaluates multimodal large language models on hierarchical and holistic video understanding, specifically probing temporal reasoning, countercommonsense comprehension, trajectory state tracking, and first-person streaming video analysis.

Datasets

• H²VU — total ?; splits: test (-1); repo https://github.com/siriusrecco/H2VU-BenchMark

Metrics

• accuracy (primary) — range: [0, 1]
  • Computed by comparing the model's generated answer against the ground truth label for each question. A match yields 1, otherwise 0. The final score is the average across all instances.

Input / output format

Input: A video clip (processed via uniform frame sampling or fixed frame-rate sampling) paired with a natural language question or instruction.

Output: A natural language text response containing the model's predicted answer.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, ground_truths):
    if normalize(pred) == normalize(gold):
        correct += 1
    total += 1
return correct / total

Common pitfalls

• Sparse frame sampling (e.g., 32 frames) often misses fine-grained motion dynamics and inter-frame dependencies, unfairly penalizing models on temporal tracking tasks.
• Models frequently over-rely on textual priors or real-world assumptions rather than observing actual visual cues, leading to systematic errors in countercommonsense tasks.
• Performance drops significantly in online/streaming settings compared to offline videos for models not explicitly optimized for continuous frame processing.

Evidence (verbatim from paper)

We utilized accuracy as the evaluation metric, determined by comparing the model outputs with the ground truth, without involving any third-party models.

Citation

@misc{wu2025h2vubenchmark,
  title={H2VU-Benchmark: A Comprehensive Benchmark for Hierarchical Holistic Video Understanding},
  author={Qi Wu et al. (2025)},
  year={2025},
  note={arXiv:2503.24008}
}

• arXiv: 2503.24008