# Save Video Text Retrieval Eval

> Evaluates a model's ability to retrieve relevant videos given a natural language query in an audio-visual setting. It specifically probes how well speech-aware representations and early vision-audio alignment improve cross-modal matching accuracy across diverse video-text benchmarks. Use when the user wants to benchmark on MSRVTT-9k, MSRVTT-7k, VATEX, Charades, LSMDC, or asks about evaluating this task. Reports SumR.

- Skill: `qhjqhj00/save-video-text-retrieval-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/save-video-text-retrieval-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/save-video-text-retrieval-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/save-video-text-retrieval-eval

---


# save-video-text-retrieval-eval

> SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval — Zhao et al. (2026) (arXiv:2603.08224, 2026)

## What this evaluates

Evaluates a model's ability to retrieve relevant videos given a natural language query in an audio-visual setting. It specifically probes how well speech-aware representations and early vision-audio alignment improve cross-modal matching accuracy across diverse video-text benchmarks.

## Datasets

- **MSRVTT-9k** — total 10000; splits: train (9000), test (1000)
- **MSRVTT-7k** — total 10000; splits: train (6513), val (497), test (2990)
- **VATEX** — total 28991; splits: train (25991), val (1500), test (1500)
- **Charades** — total 9848; splits: train (7985), test (1863)
- **LSMDC** — total 109487; splits: train (101079), val (7408), test (1000)

## Metrics

- `R@k (R1, R5, R10)` — range: percent
  - Recall at top-k: fraction of ground-truth videos retrieved within the top-k ranked results for each query.
- `SumR` **(primary)** — range: percent
  - Sum of R1, R5, and R10 scores (R1 + R5 + R10). Used as the overall performance score.
- `mR1` — range: percent
  - Mean R1 score averaged across all evaluated datasets.

## Input / output format

**Input**: A natural language query (caption) paired with a video containing visual frames and audio/speech transcripts.

**Output**: A ranked list of video candidates or similarity scores for each query, sorted by relevance.

## Scoring recipe

```python
def compute_metrics(predictions, gold_indices):
    recalls = {}
    for k in [1, 5, 10]:
        hits = sum(1 for pred in predictions if pred[:k] in gold_indices)
        recalls[f'R{k}'] = hits / len(gold_indices) * 100
    sumr = sum(recalls[f'R{k}'] for k in [1, 5, 10])
    return recalls, sumr
```

## Common pitfalls

- MSRVTT-9k and Charades lack validation splits, so the best checkpoint is selected based on peak test performance rather than validation R1.
- VATEX uses a specific partition from Chen et al. rather than the official default, requiring strict adherence to the cited split.
- SumR is reported as a sum of percentages (e.g., ~216), not an average, which can cause confusion when comparing against single R@k values.

## Evidence (verbatim from paper)

> Evaluation criteria. We report standard rank-based retrieval metrics, i.e. Recall at top $k$ (${1,5,10}$), and SumR (R1+R5+R10) as an overall score.

## Citation

```bibtex
@misc{zhao2026save,
  title={SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2603.08224}
}
```

- arXiv: 2603.08224