Mmbe Eval

mmbe-eval

VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks — Jiang et al. (2024) (arXiv:2410.05160, 2024)

What this evaluates

Evaluates the ability of vision-language models to generate unified multimodal embeddings for diverse tasks including classification, visual question answering, retrieval, and visual grounding. It probes zero-shot generalization to unseen datasets and the model's capacity to follow task-specific instructions for cross-modal alignment.

Datasets

• MMEB — total ?; splits: train (662000), test (-1)

Metrics

• Precision@1 (primary) — range: percent
  • Measures the ratio of positive candidates being ranked in the top place for all queries.

Input / output format

Input: Image and text pairs (query and target), optionally accompanied by task-specific instructions.

Output: Fixed-dimensional embedding vectors representing the image-text combination.

Scoring recipe

def precision_at_1(predictions, gold):
    correct = 0
    for pred, gold_idx in zip(predictions, gold):
        if pred[0] == gold_idx:
            correct += 1
    return correct / len(gold)

Common pitfalls

• Long text queries or targets may be truncated due to the text encoder's length limitations in baseline models.
• Instruction usage significantly impacts performance differently for CLIP vs VLM2Vec backbones, so results must be reported with and without instructions.
• Shallow fusion baselines (e.g., score-level addition) do not leverage deep cross-modal integration, making direct comparison with deep-fusion models potentially unfair without careful configuration.

Evidence (verbatim from paper)

We report Precision@1 for all models in Table [2]. It measures the ratio of positive candidates being ranked in the top place for all queries.

Citation

@misc{jiang2024vlm2vec,
  title={VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2410.05160}
}

• arXiv: 2410.05160