Lovr Eval

lovr-eval

LoVR: A Benchmark for Long Video Retrieval in Multimodal Contexts — Cai et al. (2025) (arXiv:2505.13928, 2025)

What this evaluates

Evaluates a model's ability to retrieve relevant long-form videos or fine-grained clips based on rich, narrative-driven text queries. It probes temporal reasoning, semantic alignment across extended durations, and robustness to long-context inputs and varying frame sampling strategies.

Datasets

• LoVR — total 467; splits: test (-1); repo https://github.com/TechNomad-ds/LoVR-benchmark

Metrics

• Recall@K (primary) — range: [0, 1]
  • Proportion of true relevant items among the top-K retrieved results. Computed for K=1, 5, and 10. Averaged across all query-video/clip pairs.
• Retrieval Time — range: ms
  • End-to-end latency measured in milliseconds. Reported under two configurations: with I/O (including disk read/write and network transmission) and without I/O (pure computation like vector similarity calculation and ranking).

Input / output format

Input: A video (full-length or fine-grained clip) represented as a sequence of sampled frames, paired with a text query/caption.

Output: A ranked list of video or clip identifiers sorted by similarity score to the text query.

Scoring recipe

def compute_recall_at_k(retrieved_ids, ground_truth_id, k):
    top_k = retrieved_ids[:k]
    return 1.0 if ground_truth_id in top_k else 0.0

def compute_recall_at_k_avg(retrieved_ids_list, ground_truth_ids_list, k):
    scores = [compute_recall_at_k(r, g, k) for r, g in zip(retrieved_ids_list, ground_truth_ids_list)]
    return sum(scores) / len(scores) if scores else 0.0

Common pitfalls

• Using image encoders without averaging frame-level embeddings to produce a single video-level representation before similarity computation.
• Confusing 'Time w/ IO' (real-world deployment bottleneck) with 'Time w/o IO' (pure algorithmic computation potential).
• Assuming increasing sampled frames linearly improves accuracy; the benchmark shows diminishing returns or added noise beyond 10-20 frames.
• Failing to process long query contexts end-to-end, causing models to ignore critical details at the end of the prompt.

Evidence (verbatim from paper)

This metric is measured using Recall@K, which quantifies the proportion of true relevant items among the top-K retrieved results and reflects the recall capability of the retrieval model.

Citation

@misc{cai2025lovr,
  title={LoVR: A Benchmark for Long Video Retrieval in Multimodal Contexts},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2505.13928}
}

• arXiv: 2505.13928