qhjqhj00 mlvu-eval
From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents — Niu Lian et al. (2026) (arXiv:2603.01455, 2026)
What this evaluates
Evaluates long-horizon video understanding and reasoning capabilities of multimodal models on extended video sequences.
Datasets
- MLVU — total ?; splits: test (-1)
Metrics
M-Avg(primary) — range: percent- Macro-average accuracy across multiple video understanding tasks or categories in the MLVU benchmark.
Input / output format
Input: Long-horizon video clips.
Output: Textual answers to video-based questions (VQA).
Scoring recipe
def score(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
return (correct / len(gold)) * 100
Common pitfalls
- Performance is highly sensitive to subtitle availability (w/ vs w/o settings reported separately).
- Evaluation spans short, medium, and long video durations, requiring models to handle varying temporal contexts.
Evidence (verbatim from paper)
Table 1: Comparison on two long-video understanding benchmarks: Video-MME and MLVU. For Video-MME, we report results under both w/ and w/o subtitle settings (w/ = with subtitles; w/o = without subtitles). For MLVU, we report M-Avg.
Citation
@misc{lian2026mmmem,
title={From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents},
author={Niu Lian et al. (2026)},
year={2026},
note={arXiv:2603.01455}
}
- arXiv: 2603.01455