qhjqhj00 video-mme-eval
From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents — Niu Lian et al. (2026) (arXiv:2603.01455, 2026)
What this evaluates
Evaluates long-horizon video understanding and reasoning capabilities of multimodal models on extended video sequences.
Datasets
- Video-MME — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered video-based questions. Reported separately for short, medium, and long video durations, as well as an overall average.
Input / output format
Input: Long-horizon video clips, optionally accompanied by subtitles or captions.
Output: Textual answers to video-based questions (VQA).
Scoring recipe
def score(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
return (correct / len(gold)) * 100
Common pitfalls
- Performance is highly sensitive to subtitle availability (w/ vs w/o settings reported separately).
- Evaluation spans short, medium, and long video durations, requiring models to handle varying temporal contexts.
Evidence (verbatim from paper)
Table 1: Comparison on two long-video understanding benchmarks: Video-MME and MLVU. For Video-MME, we report results under both w/ and w/o subtitle settings (w/ = with subtitles; w/o = without subtitles).
Citation
@misc{lian2026mmmem,
title={From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents},
author={Niu Lian et al. (2026)},
year={2026},
note={arXiv:2603.01455}
}
- arXiv: 2603.01455