Mmeb Eval

mmeb-eval

mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data — Chen et al. (2025) (arXiv:2502.08468, 2025)

What this evaluates

Evaluates the cross-modal alignment and generalization capabilities of multimodal embedding models across classification, VQA, retrieval, and visual grounding tasks.

Datasets

• MMEB — total ?; splits: test (-1)

Metrics

• Precision@1 (primary) — range: [0, 1]
  • The fraction of queries where the top-1 retrieved image matches the ground truth label or description.
• Recall@10 — range: [0, 1]
  • The fraction of queries where the ground truth image appears in the top-10 retrieved results.

Input / output format

Input: Image and corresponding text query/prompt for each task type (classification, VQA, retrieval, visual grounding).

Output: Dense embedding vectors for the image and text, used for similarity matching.

Scoring recipe

def compute_precision_at_1(img_embs, txt_embs, labels):
    sims = cosine_similarity(img_embs, txt_embs)
    preds = argmax(sims, axis=1)
    return mean(preds == labels)

def compute_recall_at_10(img_embs, txt_embs, labels):
    sims = cosine_similarity(img_embs, txt_embs)
    top_k = argsort(sims, axis=1, descending=True)[:, :10]
    return mean(labels in top_k)

Common pitfalls

• Distinguishing between zero-shot evaluation (using only synthetic data) and supervised evaluation (using in-distribution training data) is critical for fair comparison.
• The synthetic dataset distribution is intentionally skewed 1:1:2 for classification:VQA:retrieval, which affects generalization compared to models trained on pure retrieval data.

Evidence (verbatim from paper)

We evaluate the general embedding performance in terms of Precision@1 on the MMEB benchmarkJiang et al. ([2024b]). This benchmark comprises 36 multimodal embedding tasks across four categories: classification (10), VQA (10), retrieval (12), and visual grounding (4). ... we conduct experiments on seven languages of XTD and report Recall@10 results.

Citation

@misc{chen2025mme5,
  title={mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2502.08468}
}

• arXiv: 2502.08468