# Mmbe Eval

> Evaluates the ability of vision-language models to generate unified multimodal embeddings for diverse tasks including classification, visual question answering, retrieval, and visual grounding. It probes zero-shot generalization to unseen datasets and the model's capacity to follow task-specific instructions for cross-modal alignment. Use when the user wants to benchmark on MMEB, or asks about evaluating this task. Reports Precision@1.

- Skill: `qhjqhj00/mmbe-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mmbe-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mmbe-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mmbe-eval

---


# mmbe-eval

> VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks — Jiang et al. (2024) (arXiv:2410.05160, 2024)

## What this evaluates

Evaluates the ability of vision-language models to generate unified multimodal embeddings for diverse tasks including classification, visual question answering, retrieval, and visual grounding. It probes zero-shot generalization to unseen datasets and the model's capacity to follow task-specific instructions for cross-modal alignment.

## Datasets

- **MMEB** — total ?; splits: train (662000), test (-1)

## Metrics

- `Precision@1` **(primary)** — range: percent
  - Measures the ratio of positive candidates being ranked in the top place for all queries.

## Input / output format

**Input**: Image and text pairs (query and target), optionally accompanied by task-specific instructions.

**Output**: Fixed-dimensional embedding vectors representing the image-text combination.

## Scoring recipe

```python
def precision_at_1(predictions, gold):
    correct = 0
    for pred, gold_idx in zip(predictions, gold):
        if pred[0] == gold_idx:
            correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Long text queries or targets may be truncated due to the text encoder's length limitations in baseline models.
- Instruction usage significantly impacts performance differently for CLIP vs VLM2Vec backbones, so results must be reported with and without instructions.
- Shallow fusion baselines (e.g., score-level addition) do not leverage deep cross-modal integration, making direct comparison with deep-fusion models potentially unfair without careful configuration.

## Evidence (verbatim from paper)

> We report Precision@1 for all models in Table [2]. It measures the ratio of positive candidates being ranked in the top place for all queries.

## Citation

```bibtex
@misc{jiang2024vlm2vec,
  title={VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2410.05160}
}
```

- arXiv: 2410.05160