Lvsum Eval

lvsum-eval

LVSum: A Benchmark for Timestamp-Aware Long Video Summarization — Patel et al. (2026) (arXiv:2604.10024, 2026)

What this evaluates

This benchmark evaluates multimodal large language models' ability to perform timestamp-aware summarization of long videos. It probes temporal grounding, instruction adherence regarding length constraints, and cross-modal consistency between visual/audio content and generated text descriptions.

Datasets

• LVSum — total 72; splits: test (72)

Metrics

• Kendall's tau & Spearman's rho (primary) — range: [-1, 1]
  • Rank correlation coefficients measuring the correspondence between predicted and ground-truth importance rankings at second-level granularity.
• Content Relevance (CR) — range: other
  • LLM-as-judge metric scoring semantic alignment between the generated summary and ground-truth key moments on a 1-5 scale.
• Modality Coherence (MC) — range: other
  • LLM-as-judge metric scoring cross-modal consistency (visual, auditory, textual alignment) for each predicted segment on a 1-5 scale.

Input / output format

Input: 96 uniformly sampled video frames, a timestamped transcript of spoken narration/dialogue, and a system prompt instructing the model to generate a summary covering ~15% of the video duration.

Output: A list of timestamped segments with textual descriptions, constrained to approximately 15% of the total video length.

Scoring recipe

# 1. Temporal Grounding
pred_ranks = rank_segments_by_importance(predictions)
gold_ranks = rank_segments_by_importance(gold)
tau = kendalltau(pred_ranks, gold_ranks)
rho = spearmanr(pred_ranks, gold_ranks)

# 2. LLM-as-Judge Metrics (CR & MC)
cr_scores, mc_scores = [], []
for seg in predictions:
    visual_audio = extract_media(seg.start, seg.end)
    judge_output = llm_judge_prompt(seg.text, visual_audio, gold)
    cr_scores.append(parse_score(judge_output, scale=5))
    mc_scores.append(parse_score(judge_output, scale=5))
avg_cr = mean(cr_scores)
avg_mc = mean(mc_scores)

Common pitfalls

• Models frequently exceed the 15% duration budget despite explicit instructions, skewing temporal ranking metrics.
• Using frame-level granularity for importance scoring instead of second-level granularity misaligns with MLLM timestamp precision.
• LLM-as-judge metrics depend heavily on the specific judge model and prompt templates, which may introduce scoring bias.

Evidence (verbatim from paper)

In practice, current MLLMs are prone to producing summary segments whose accumulated duration exceeds the target 15% budget, even when explicitly instructed with a length constraint. Moreover,[otani2019rethinking] showed that Kendall’s $ au$ and Spearman’s $ ho$ are more robust than F1-based metrics when segment lengths vary. For these reasons, we use Kendall’s $ au$ and Spearman’s $ ho$ as our primary metrics and report the average predicted summary length to contextualize metric values.

Citation

@misc{patel2026lvsum,
  title={LVSum: A Benchmark for Timestamp-Aware Long Video Summarization},
  author={Patel et al. (2026)},
  year={2026},
  note={arXiv:2604.10024}
}

• arXiv: 2604.10024