Lvbench Eval

lvbench-eval

LVBench: An Extreme Long Video Understanding Benchmark — Wang et al. (2024) (arXiv:2406.08035, 2024)

What this evaluates

This benchmark evaluates multimodal models' ability to comprehend extreme-length videos (averaging ~70 minutes) by testing six core temporal understanding capabilities. It probes long-term memory, multi-hop reasoning, and instruction-following across diverse video categories like sports, documentaries, and TV shows.

Datasets

• LVBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions, calculated as (correct predictions / total questions) × 100.

Input / output format

Input: Video frames sampled at 1 FPS (or fixed counts like 32/96) concatenated with a text prompt containing the question and four multiple-choice options (A-D), followed by the instruction: 'Please select the best answer from the options above and directly provide the letter representing your choice without giving any explanation.'

Output: A single uppercase letter (A, B, C, or D) corresponding to the selected option.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_letter = extract_answer_letter(pred)
        if pred_letter == gold:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Models frequently violate the strict output constraint, generating phrases like 'None of the above' instead of the required letter, necessitating robust extraction pipelines.
• Performance is highly sensitive to frame sampling density; sparse sampling yields near-random results, while 1 FPS is required to capture transient visual cues.
• Certain models exhibit strong response biases (e.g., defaulting to option A), which can artificially inflate accuracy if not accounted for during evaluation.

Evidence (verbatim from paper)

humans achieve a very high accuracy of 94.4% on average, setting a strong benchmark across all categories. In contrast, the overall performance of Gemini-2.5-Pro and Seed1.5-VL was considerably lower, at 67.4% and 64.0%, respectively.

Citation

@misc{wang2024lvbench,
  title={LVBench: An Extreme Long Video Understanding Benchmark},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.08035}
}

• arXiv: 2406.08035