Sharegpt4video Eval

sharegpt4video-eval

ShareGPT4Video: Improving Video Understanding and Generation with Better Captions — Lin Chen et al. (2024) (arXiv:2406.04325, 2024)

What this evaluates

Evaluates the temporal understanding and video-language alignment capabilities of Large Video-Language Models (LVLMs) across three multi-modal video benchmarks. It probes the model's ability to answer questions about video content, track temporal changes, and comprehend complex video sequences without relying on single-frame cues.

Datasets

• VideoBench — total 15000; splits: test (15000)
• MVBench — total 4000; splits: test (4000)
• TempCompass — total 7540; splits: test (7540)

Metrics

• benchmark accuracy (VideoBench, MVBench, TempCompass) (primary) — range: percent
  • Percentage of correctly answered questions across multiple-choice, yes/no, and open-ended QA formats. Evaluated using exact match for objective questions and LLM-as-a-judge or human evaluation for open-ended questions, as per each benchmark's official protocol.

Input / output format

Input: Video frames (uniformly sampled 16 frames arranged in a 4x4 grid) concatenated with a text question or instruction.

Output: Text answer generated by the model.

Scoring recipe

For each benchmark, follow its official evaluation script. Typically:
1. Parse model predictions.
2. For MC/YesNo: compute exact match accuracy against gold labels.
3. For open-ended: use an LLM judge or human annotators to score correctness.
4. Average scores across all question types and report as percentage.

Common pitfalls

• Benchmarks combine heterogeneous question types (MC, Yes/No, open-ended) that require different evaluation metrics; averaging them without weighting can mask performance on specific capabilities.
• Temporal benchmarks like TempCompass explicitly test for frame-level temporal reasoning; models often cheat by relying on single-frame visual cues or video metadata, leading to inflated scores if not strictly controlled.

Evidence (verbatim from paper)

To thoroughly explore the benefits that our high-quality video-caption data bring to LVLMs, we conduct comprehensive evaluations of the model across three multi-modal video benchmarks. VideoBench [[44]] curates approximately 15,000 QA pairs spanning 10 evaluation dimensions from 13 existing data sources, such as MSVD-QA [[62]], MSRVTT-QA [[62]], Activitynet-QA [[67]], etc. MVBench [[30]] is designed to challenge LVLMs with video tasks that cannot be effectively resolved by single-frame reliance, featuring 4,000 QA pairs derived from 11 public video benchmarks. TempCompass [[38]] specifically assesses the nuanced performance of LVLMs across various temporal aspects, such as speed, direction, and attribute changes. It includes 410 videos and 7,540 meticulously collected instructions, emphasizing temporal comprehension and interaction.

Citation

@misc{chen2024sharegpt4video,
  title={ShareGPT4Video: Improving Video Understanding and Generation with Better Captions},
  author={Lin Chen et al. (2024)},
  year={2024},
  note={arXiv:2406.04325}
}

• arXiv: 2406.04325