Egmm Corpus Eval

egmm-corpus-eval

EgMM-Corpus: A Multimodal Vision-Language Dataset for Egyptian Culture — Gamil et al. (2025) (arXiv:2510.16198, 2025)

What this evaluates

Probes zero-shot visual-language alignment and cultural recognition capabilities of vision-language models on Egyptian cultural concepts. It measures how well models can classify images into specific cultural categories and retrieve matching text descriptions without fine-tuning.

Datasets

• EgMM-Corpus — total 3000; splits: test (995)

Metrics

• Acc@1 (primary) — range: percent
  • Top-1 accuracy: the fraction of images where the correct cultural concept name is ranked first among all candidate labels.
• Acc@5 — range: percent
  • Top-5 accuracy: the fraction of images where the correct concept appears within the top five predicted labels.
• I2T R@1 — range: percent
  • Image-to-Text Recall@1: the fraction of image queries where the correct text description is ranked first.
• T2I R@1 — range: percent
  • Text-to-Image Recall@1: the fraction of text queries where the correct image is ranked first.
• I2T R@5 — range: percent
  • Image-to-Text Recall@5: the fraction of image queries where the correct text description appears in the top five results.

Input / output format

Input: For classification: an image and a fixed set of candidate text labels (cultural concept names). For retrieval: an image paired with all text descriptions in the corpus, or a text query paired with all images.

Output: A ranked list of candidate labels (classification) or a ranked list of images/texts (retrieval) based on model similarity scores.

Scoring recipe

# Classification
acc1 = sum(1 for pred, gold in zip(preds, golds) if pred == gold) / len(golds)
acc5 = sum(1 for top5, gold in zip(top5_preds, golds) if gold in top5) / len(golds)

# Retrieval (I2T)
r1 = sum(1 for img, gold_txt in zip(images, gold_texts) if gold_txt in rank_texts(img, k=1)) / len(gold_texts)
r5 = sum(1 for img, gold_txt in zip(images, gold_texts) if gold_txt in rank_texts(img, k=5)) / len(gold_texts)

Common pitfalls

• The evaluation uses a strict zero-shot setting, so low scores reflect pre-training data gaps rather than model architecture flaws or lack of fine-tuning.
• Retrieval metrics are evaluated in both directions (I2T and T2I), which often yield asymmetric scores due to vocabulary-image distribution mismatches.
• The reported 995-image sample is a subset of the full 3,000+ corpus, so results may not represent the complete dataset's difficulty or distribution.

Evidence (verbatim from paper)

For zero-shot classification, each image is matched against all concept names, and we report Top-1 and Top-5 accuracy. For retrieval, we adopt Recall@K (R@K)-a standard measure in cross-modal learning-evaluated in both directions: Image-to-Text (I2T): each image query is used to rank all textual descriptions. R@K represents the fraction of images for which the correct text appears among the top K retrieved results. Text-to-Image (T2I): each textual concept query is used to rank all images. R@K measures the percentage of times the correct image is found within the top K results. We primarily report R@1 (strict correctness) and R@5 (top-5 relevance) to provide a comprehensive view of CLIP’s visual–textual alignment in a zero-shot setting.

Citation

@misc{gamil2025egmmcorpus,
  title={EgMM-Corpus: A Multimodal Vision-Language Dataset for Egyptian Culture},
  author={Gamil et al. (2025)},
  year={2025},
  note={arXiv:2510.16198}
}

• arXiv: 2510.16198