Longvideo Bench Eval

longvideo-bench-eval

LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding — Wu et al. (2024) (arXiv:2407.15754, 2024)

What this evaluates

This benchmark evaluates long-context video-language understanding by testing a model's ability to retrieve specific moments from lengthy videos and reason over multimodal details. It distinguishes between single-moment visual perception and multi-moment relational reasoning across 17 fine-grained categories.

Datasets

• LongVideoBench — total 6678; splits: test (6678)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of questions where the model's predicted option exactly matches the ground-truth correct answer.

Input / output format

Input: A long video (up to 60 minutes) with interleaved frames and subtitles, a referring query that identifies specific moments, and a multiple-choice question requiring reasoning over those moments.

Output: The model must select the correct answer option from the provided multiple-choice list.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    return correct / len(golds)

Common pitfalls

• Models may exploit single-frame bias or short-context shortcuts instead of truly reasoning over long sequences.
• Confusion between L1 (single-moment perception) and L2 (multi-moment relational) tasks can skew category-level performance analysis.
• Subtitle-based referring queries may be answered via text-only pathways, bypassing actual video understanding.

Evidence (verbatim from paper)

We employ the multiple-choice question format, where several distracting options are provided alongside the correct answer option.

Citation

@misc{wu2024longvideobench,
  title={LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding},
  author={Wu et al. (2024)},
  year={2024},
  note={arXiv:2407.15754}
}

• arXiv: 2407.15754