# Mlvu Eval

> Evaluates long-horizon video understanding and reasoning capabilities of multimodal models on extended video sequences. Use when the user wants to benchmark on MLVU, or asks about evaluating this task. Reports M-Avg.

- Skill: `qhjqhj00/mlvu-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mlvu-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mlvu-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mlvu-eval

---


# mlvu-eval

> From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents — Niu Lian et al. (2026) (arXiv:2603.01455, 2026)

## What this evaluates

Evaluates long-horizon video understanding and reasoning capabilities of multimodal models on extended video sequences.

## Datasets

- **MLVU** — total ?; splits: test (-1)

## Metrics

- `M-Avg` **(primary)** — range: percent
  - Macro-average accuracy across multiple video understanding tasks or categories in the MLVU benchmark.

## Input / output format

**Input**: Long-horizon video clips.

**Output**: Textual answers to video-based questions (VQA).

## Scoring recipe

```python
def score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Performance is highly sensitive to subtitle availability (w/ vs w/o settings reported separately).
- Evaluation spans short, medium, and long video durations, requiring models to handle varying temporal contexts.

## Evidence (verbatim from paper)

> Table 1: Comparison on two long-video understanding benchmarks: Video-MME and MLVU. For Video-MME, we report results under both w/ and w/o subtitle settings (w/ = with subtitles; w/o = without subtitles). For MLVU, we report M-Avg.

## Citation

```bibtex
@misc{lian2026mmmem,
  title={From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents},
  author={Niu Lian et al. (2026)},
  year={2026},
  note={arXiv:2603.01455}
}
```

- arXiv: 2603.01455