M Beir Eval

m-beir-eval

UniIR: Training and Benchmarking Universal Multimodal Information Retrievers — Wei et al. (2023) (arXiv:2311.17136, 2023)

What this evaluates

Evaluates multimodal information retrieval models across eight heterogeneous query-to-candidate modalities (text, image, image-text pairs) using instruction-tuned and fine-tuned vision-language models. Probes zero-shot generalization, cross-modality alignment, and the impact of instruction tuning on retrieval accuracy in large-scale candidate pools.

Datasets

• M-BEIR — total 5600000; splits: train (-1), test (-1)

Metrics

• Recall@5 (primary) — range: [0, 1]
  • Fraction of queries where the ground-truth candidate appears in the top-5 retrieved results. For Fashion200K and FashionIQ, Recall@10 is reported instead.

Input / output format

Input: Query consisting of text, image, or image-text pair, accompanied by a task-specific instruction string. Candidate pool containing images, texts, or image-text pairs.

Output: Ranked list of candidate IDs/embeddings, typically returning the top-5 (or top-10 for specific datasets) retrieved candidates.

Scoring recipe

def recall_at_k(predictions, gold, k=5):
    top_k = set(predictions[:k])
    return len(top_k.intersection(gold)) / len(gold)

Common pitfalls

• Zero-shot models fail on heterogeneous global pools without task instructions, often retrieving wrong modalities.
• Score-level fusion (CLIP) generally outperforms feature-level fusion (BLIP) for instruction-tuned models.
• Recall@10 is used instead of Recall@5 for Fashion200K and FashionIQ datasets.

Evidence (verbatim from paper)

We report the main results on M-BEIR in Table 2, where models retrieve candidates from the 5.6M pool. ... The average Recall@5 has increased by 12.8 and 10.9, respectively.

Citation

@misc{wei2023uniir,
  title={UniIR: Training and Benchmarking Universal Multimodal Information Retrievers},
  author={Wei et al. (2023)},
  year={2023},
  note={arXiv:2311.17136}
}

• arXiv: 2311.17136