Multimodal Retrieval Eval

multimodal-retrieval-eval

Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval — Kong et al. (2025) (arXiv:2505.19650, 2025)

What this evaluates

Evaluates fine-grained and coarse-grained cross-modal retrieval capabilities across text, image, and video modalities. It probes a model's ability to align and retrieve relevant visual or textual content given a query from a different modality, including instruction-based queries.

Datasets

• CaReBench — total ?; splits: test (-1)
• ShareGPT4V — total ?; splits: test (-1)
• Urban1K — total ?; splits: test (-1)
• DOCCI — total ?; splits: test (-1)
• WebVid-CoVR — total ?; splits: test (-1)
• MMEB — total ?; splits: test (-1)
• Flickr30K — total ?; splits: test (-1)
• MSR-VTT — total ?; splits: test (-1)
• MSVD — total ?; splits: test (-1)
• DiDeMo — total ?; splits: test (-1)

Metrics

• Recall@1 (primary) — range: percent
  • The fraction of queries for which the ground-truth matching item appears in the top-1 retrieved result.
• Recall@5 — range: percent
  • The fraction of queries for which the ground-truth matching item appears in the top-5 retrieved results.
• Recall@10 — range: percent
  • The fraction of queries for which the ground-truth matching item appears in the top-10 retrieved results.
• Recall@50 — range: percent
  • The fraction of queries for which the ground-truth matching item appears in the top-50 retrieved results.
• Average Score — range: percent
  • The arithmetic mean of in-distribution (IND) and out-of-distribution (OOD) scores across meta-tasks on the MMEB benchmark.

Input / output format

Input: Text queries paired with candidate images or videos, or visual queries paired with candidate text descriptions.

Output: A ranked list of retrieved candidates or similarity scores for each query-candidate pair.

Scoring recipe

def recall_at_k(retrieved_indices, gold_index, k):
    top_k = retrieved_indices[:k]
    return 1.0 if gold_index in top_k else 0.0
# Average across all queries in the test set

Common pitfalls

• Confusing text-to-visual (T→V) versus visual-to-text (V→T) retrieval directions, which often show asymmetric performance.
• Failing to distinguish between fine-grained (e.g., CaReBench) and coarse-grained (e.g., MSR-VTT) tasks, as fine-grained benchmarks demand significantly higher precision.
• Reporting only R@1 when the protocol explicitly notes that R@5, R@10, and R@50 are also standard for robustness evaluation.

Evidence (verbatim from paper)

All experimental results are reported in Recall@1 unless otherwise specified.

Citation

@misc{kong2025modality,
  title={Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval},
  author={Kong et al. (2025)},
  year={2025},
  note={arXiv:2505.19650}
}

• arXiv: 2505.19650