Mieb Eval

mieb-eval

MIEB: Massive Image Embedding Benchmark — Xiao et al. (2025) (arXiv:2504.10471, 2025)

What this evaluates

MIEB evaluates the diverse capabilities of image and image-text embedding models across 130 tasks spanning retrieval, document understanding, classification, clustering, compositionality, and visual question answering. It probes zero-shot generalization, multilingual understanding, spatial/depth reasoning, and the model's ability to encode visual representations of text.

Datasets

• MIEB — total ?; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It measures the quality of a ranked list of retrieved items by comparing the actual relevance scores to an ideal sorted list, discounting lower-ranked relevant items logarithmically.
• accuracy — range: [0, 1]
  • The proportion of correctly predicted class labels or top-1 matches between image embeddings and text prompt embeddings.
• Normalized Mutual Information (NMI) — range: [0, 1]
  • A clustering evaluation metric that measures the agreement between predicted cluster assignments and ground truth labels, normalized by the entropy of both distributions.
• Spearman correlation — range: [-1, 1]
  • A rank-based correlation coefficient measuring the monotonic relationship between predicted embedding similarity scores and human annotations.

Input / output format

Input: Pairs or sets of images and texts (queries and documents), which may be interleaved or multilingual. For classification/clustering tasks, images with class labels or text prompts are provided.

Output: Fixed-dimensional embedding vectors for each image and text instance. For classification tasks, predicted class labels or similarity scores between image embeddings and text prompt embeddings.

Scoring recipe

def compute_ndcg_at_k(gold_relevance, k=10):
    import math
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(gold_relevance[:k]))
    ideal = sorted(gold_relevance, reverse=True)[:k]
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

# Usage: predictions = model.encode(query), gold = relevance_scores
# metric_value = compute_ndcg_at_k(predictions, k=10)

Common pitfalls

• Models trained with non-representation losses (e.g., autoregressive models) often lack good off-the-shelf zero-shot performance but may still perform well in linear probing.
• Visual STS tasks primarily measure OCR capabilities of vision encoders, which are often overlooked when evaluating semantic similarity.
• Linear probing uses a fixed few-shot setting (16 shots per class) rather than full-dataset training, which can affect performance trends compared to standard classification benchmarks.

Evidence (verbatim from paper)

We use nDCG@10 as the primary metric, and recall@1/map@5 for some tasks to align with prior work or adjust for difficulty.

Citation

@misc{xiao2025mieb,
  title={MIEB: Massive Image Embedding Benchmark},
  author={Xiao et al. (2025)},
  year={2025},
  note={arXiv:2504.10471}
}

• arXiv: 2504.10471