qhjqhj00 textvr-retrieval-eval
A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension — Wu et al. (2023) (arXiv:2305.03347, 2023)
What this evaluates
Evaluates cross-modal video retrieval models that must jointly process visual context and scene text (OCR tokens) to match sentence queries with relevant videos. Probes the model's ability to read, comprehend, and align fine-grained text semantics with visual frames in real-world scenarios.
Datasets
- TextVR — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/callsys/TextVR
Metrics
R@K (Recall@K)(primary) — range: [0, 1]- Proportion of queries where the ground-truth video appears in the top-K retrieved results (K=1, 5, 10). Higher is better.
MdR (Median Rank)— range: rank- Median of the ranks of the ground-truth videos across all queries. Lower is better.
MnR (Mean Rank)— range: rank- Arithmetic mean of the ranks of the ground-truth videos across all queries. Lower is better.
Input / output format
Input: A sentence query, a video (represented as visual features), and scene text/OCR tokens extracted from the video frames.
Output: A ranked list of videos for each query (Language-to-Video) or a ranked list of queries for each video (Video-to-Language).
Scoring recipe
def compute_metrics(predictions, golds):
ranks = []
for pred, gold in zip(predictions, golds):
if gold in pred:
ranks.append(pred.index(gold) + 1)
else:
ranks.append(len(pred) + 1)
r1 = sum(1 for r in ranks if r <= 1) / len(ranks)
r5 = sum(1 for r in ranks if r <= 5) / len(ranks)
r10 = sum(1 for r in ranks if r <= 10) / len(ranks)
md_r = sorted(ranks)[len(ranks)//2]
mn_r = sum(ranks) / len(ranks)
return {'R@1': r1, 'R@5': r5, 'R@10': r10, 'MdR': md_r, 'MnR': mn_r}
Common pitfalls
- Confusing scene text (OCR tokens) with subtitles; the dataset specifically evaluates reading comprehension of in-video text, not closed captions.
- High false negatives in OCR detection can severely penalize retrieval performance, as missing relevant text tokens removes crucial semantic signals.
- Mean Rank (MnR) is less sensitive to top-1 improvements than R@1 or MdR due to its susceptibility to long-tail rank distributions.
Evidence (verbatim from paper)
Evaluation Metric Following previous video retrieval benchmarks, we adopt the average recall at K(R@K), median rank (MdR), and mean rank (MnR) over all queries as the metric. We consider a prediction correct if the predicted video matches the ground-truth video. Generally, the higher R@K and lower MdR, MnR show better performance.
Citation
@misc{wu2023textvr,
title={A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension},
author={Wu et al. (2023)},
year={2023},
note={arXiv:2305.03347}
}
- arXiv: 2305.03347