Mvbench Eval

mvbench-eval

MVBench: A Comprehensive Multi-modal Video Understanding Benchmark — Kunchang Li et al. (2023) (arXiv:2311.17005, 2023)

What this evaluates

Evaluates multi-modal large language models' ability to understand video content, with a strong focus on temporal perception and static-to-dynamic task transformation across 20 diverse categories ranging from basic perception to complex reasoning.

Datasets

• MVBench — total ?; splits: test (-1); repo https://github.com/OpenGVLab/Ask-Anything

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers across all test instances. Calculated as (number of correct predictions / total number of predictions) * 100.

Input / output format

Input: 16-frame video clips paired with multiple-choice questions and system prompts.

Output: Text response containing the selected option (e.g., 'Best option: (A)') or direct answer string.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Using 4-frame inputs instead of the specified 16-frame evaluation setting.
• Failing to use the explicit answer prompt ('Best option: (') which significantly boosts option extraction precision.
• Confusing training instruction data sizes (e.g., 1.1M) with the test set size.
• Evaluating on static image tasks without applying the static-to-dynamic transformation methodology.

Evidence (verbatim from paper)

For evaluation, we input 16-frame videos with elaborate prompts for better results. VideoChat [42], a top-performing video MLLM, only marginally surpasses VideoChat2_text by 0.8% in average accuracy (35.5% vs. 34.7%), with the latter generating responses from text alone.

Citation

@misc{li2023mvbench,
  title={MVBench: A Comprehensive Multi-modal Video Understanding Benchmark},
  author={Kunchang Li et al. (2023)},
  year={2023},
  note={arXiv:2311.17005}
}

• arXiv: 2311.17005