Mmmeb Eval

mmmeb-eval

xVLM2Vec: Adapting LVLM-based embedding models to multilinguality using Self-Knowledge Distillation — Musacchio et al. (2025) (arXiv:2503.09313, 2025)

What this evaluates

Evaluates cross-lingual and cross-modal embedding alignment for image-text retrieval, classification, visual question answering, and visual grounding. Probes whether multilingual adaptation preserves semantic consistency across languages and modalities without degrading English performance.

Datasets

• MMMEB — total ?; splits: test (-1); repo https://github.com/swapUniba/xVLM2Vec

Metrics

• P@1 (primary) — range: percent
  • Precision at 1: for each query, similarity scores are computed between the query embedding and all candidate target embeddings. The metric returns 1 if the ground-truth target is ranked first, and 0 otherwise. Results are averaged across queries and reported as AVG-3 (I2T, T2I, C) or AVG (all tasks).

Input / output format

Input: Multimodal inputs formatted as specific instruction prompts. Images are represented via the special token <$|$image_1$|$> within text strings. Queries and targets follow task-specific templates (e.g., I2T: <$|$image_1$|$>\nFind an image caption...). Plain formatting removes trailing punctuation; punctuation formatting adds a period or question mark.

Output: A dense embedding vector extracted from the last token of the model's output sequence (not the special EOS token).

Scoring recipe

def compute_p1(query_emb, target_embs, correct_idx):
    sims = cosine_similarity(query_emb, target_embs)
    rank = argsort(descending=True)[correct_idx]
    return 1.0 if rank == 0 else 0.0
# Average over all queries in the dataset

Common pitfalls

• Input formatting drastically affects performance; plain vs punctuation formatting can change P@1 by 10-40 points.
• CLIP/SigLIP models cannot be evaluated on VQA/VG tasks due to missing instruction-tuning and modality constraints.
• Non-English tasks rely on translated prompts/targets, so translation quality directly impacts scores.
• Embeddings are extracted from the last token, not a pooled or special token, making tokenization sensitive.

Evidence (verbatim from paper)

We use P@1 as the evaluation metric, following the same setup used in the MMEB benchmark. We report two averages, one for the tasks that are covered by all models (AVG-3), that are I2T, T2I and C, and one for all tasks (AVG).

Citation

@misc{musacchio2025xvlm2vec,
  title={xVLM2Vec: Adapting LVLM-based embedding models to multilinguality using Self-Knowledge Distillation},
  author={Musacchio et al. (2025)},
  year={2025},
  note={arXiv:2503.09313}
}

• arXiv: 2503.09313