Save Video Text Retrieval Eval

save-video-text-retrieval-eval

SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval — Zhao et al. (2026) (arXiv:2603.08224, 2026)

What this evaluates

Evaluates a model's ability to retrieve relevant videos given a natural language query in an audio-visual setting. It specifically probes how well speech-aware representations and early vision-audio alignment improve cross-modal matching accuracy across diverse video-text benchmarks.

Datasets

• MSRVTT-9k — total 10000; splits: train (9000), test (1000)
• MSRVTT-7k — total 10000; splits: train (6513), val (497), test (2990)
• VATEX — total 28991; splits: train (25991), val (1500), test (1500)
• Charades — total 9848; splits: train (7985), test (1863)
• LSMDC — total 109487; splits: train (101079), val (7408), test (1000)

Metrics

• R@k (R1, R5, R10) — range: percent
  • Recall at top-k: fraction of ground-truth videos retrieved within the top-k ranked results for each query.
• SumR (primary) — range: percent
  • Sum of R1, R5, and R10 scores (R1 + R5 + R10). Used as the overall performance score.
• mR1 — range: percent
  • Mean R1 score averaged across all evaluated datasets.

Input / output format

Input: A natural language query (caption) paired with a video containing visual frames and audio/speech transcripts.

Output: A ranked list of video candidates or similarity scores for each query, sorted by relevance.

Scoring recipe

def compute_metrics(predictions, gold_indices):
    recalls = {}
    for k in [1, 5, 10]:
        hits = sum(1 for pred in predictions if pred[:k] in gold_indices)
        recalls[f'R{k}'] = hits / len(gold_indices) * 100
    sumr = sum(recalls[f'R{k}'] for k in [1, 5, 10])
    return recalls, sumr

Common pitfalls

• MSRVTT-9k and Charades lack validation splits, so the best checkpoint is selected based on peak test performance rather than validation R1.
• VATEX uses a specific partition from Chen et al. rather than the official default, requiring strict adherence to the cited split.
• SumR is reported as a sum of percentages (e.g., ~216), not an average, which can cause confusion when comparing against single R@k values.

Evidence (verbatim from paper)

Evaluation criteria. We report standard rank-based retrieval metrics, i.e. Recall at top $k$ (${1,5,10}$), and SumR (R1+R5+R10) as an overall score.

Citation

@misc{zhao2026save,
  title={SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2603.08224}
}

• arXiv: 2603.08224