qhjqhj00 long-video-understanding-eval
One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding — Zhang et al. (2026) (arXiv:2604.14149, 2026)
What this evaluates
Evaluates a model's ability to perform temporal reasoning and question-answering on long-duration videos (several minutes to over an hour). It probes memory retention, attention allocation across extended sequences, and the capacity to filter irrelevant visual content while preserving critical frames.
Datasets
- LongVideoBench — total ?; splits: test (-1)
- MLVU — total ?; splits: test (-1)
- VideoMME (Long) — total ?; splits: test (-1)
- LVBench — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions out of the total number of questions in the benchmark. Calculated as (correct predictions / total questions) × 100.
Input / output format
Input: Long video sequences (segmented into clips/frames) paired with a natural language question. For VideoMME, multiple-choice answer candidates are provided. Subtitles are excluded.
Output: A single predicted answer string or selected option corresponding to the question.
Scoring recipe
def compute_accuracy(predictions, gold_answers):
correct = sum(1 for p, g in zip(predictions, gold_answers) if p.strip().lower() == g.strip().lower())
return (correct / len(gold_answers)) * 100
Common pitfalls
- Position bias ('lost-in-the-middle') in long sequences can cause models to ignore mid-video content unless segmented local attention is used.
- VideoMME evaluation explicitly excludes subtitles; including them inflates performance and violates the benchmark protocol.
- Frame count limits vary by video duration (e.g., 512 vs 2048 frames), which directly impacts token compression ratios and memory constraints during inference.
Evidence (verbatim from paper)
These benchmarks adopt a question-answering format, primarily using accuracy as the evaluation metric.
Citation
@misc{zhang2026xcomp,
title={One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding},
author={Zhang et al. (2026)},
year={2026},
note={arXiv:2604.14149}
}
- arXiv: 2604.14149