Momentseeker Eval

momentseeker-eval

MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval — Yuan et al. (2025) (arXiv:2502.12558, 2025)

What this evaluates

Probes long-video moment retrieval (LVMR) capability by testing models' ability to localize specific temporal segments within long, diverse videos. It evaluates fine-grained temporal grounding and multi-modal reasoning across three semantic levels (global, event, object) using text, image, and video queries.

Datasets

• MomentSeeker — total ?; splits: test (-1)

Metrics

• R@1 (primary) — range: percent
  • Recall at 1: 1 if the top-1 predicted interval has an Intersection-over-Union (IoU) ≥ 0.3 with the ground-truth interval, else 0. Averaged across all queries.
• mAP@5 — range: percent
  • Mean Average Precision at 5: Computes precision at each rank in the top-5 predicted intervals where IoU ≥ 0.3, averages them to get AP@5, then averages AP@5 across all queries.

Input / output format

Input: Retrieval-based: video divided into fixed 10-second chunks, query (text/image/video). Generation-based: video (uniformly down-sampled frames or raw mp4), total video duration, timestamp per frame, and query prompt.

Output: Retrieval-based: ranked list of top-k video clip intervals. Generation-based: list of one or more time intervals (start/end timestamps) containing the answer.

Scoring recipe

def evaluate(pred_intervals, gt_interval, iou_thresh=0.3, k=5):
    ious = [compute_iou(p, gt_interval) for p in pred_intervals[:k]]
    recall_at_1 = 1.0 if ious[0] >= iou_thresh else 0.0
    hits = [1 if i >= iou_thresh else 0 for i in ious]
    if sum(hits) == 0:
        ap_at_5 = 0.0
    else:
        precisions = [sum(hits[:i+1]) / (i+1) for i in range(len(hits))]
        ap_at_5 = sum(p * h for p, h in zip(precisions, hits)) / sum(hits)
    return recall_at_1, ap_at_5

Common pitfalls

• IoU threshold is fixed at 0.3 for main results; changing it significantly alters scores (reported in Appendix 6.3).
• Generation-based methods exhibit strong positional bias (predicting near start/end) and degrade with longer videos due to frame downsampling, unlike retrieval methods which are position-insensitive but suffer from larger candidate pools.
• Input frame counts vary drastically across generation baselines (1 to 768 frames), making direct performance comparisons confounded by context length rather than pure model capability.

Evidence (verbatim from paper)

In the main experiment, we set the IoU threshold to 0.3. ... We adopt the same IoU-based evaluation as used for the retrieval-based methods to enable a fair comparison. Table 2: Main results across different meta-tasks. ... R@1 ... mAP@5

Citation

@misc{yuan2025momentseeker,
  title={MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval},
  author={Yuan et al. (2025)},
  year={2025},
  note={arXiv:2502.12558}
}

• arXiv: 2502.12558