M4 RAG Eval

m4-rag-eval

M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG — Anugraha et al. (2025) (arXiv:2512.05959, 2025)

What this evaluates

Evaluates vision-language models on multilingual, multicultural, and multimodal retrieval-augmented generation tasks. It measures how different retrieval strategies, language alignment, and model scale impact accuracy on culturally diverse image-question pairs.

Datasets

• CVQA — total ?; splits: test (-1)
• WorldCuisines — total ?; splits: test (-1)

Metrics

• macro-averaged accuracy (primary) — range: [0, 1]
  • Average of per-class accuracies across all datasets, computed by comparing model-generated multiple-choice answers against ground-truth labels.

Input / output format

Input: Image (I), question (q), and optional retrieved context passages (top-k=5). Prompts are provided in English or translated target languages.

Output: Multiple-choice answer selection from the provided options.

Scoring recipe

class_correct = defaultdict(int)
class_total = defaultdict(int)
for pred, gold in zip(predictions, gold_labels):
    class_total[gold] += 1
    if pred == gold:
        class_correct[gold] += 1
class_acc = [class_correct[c] / class_total[c] for c in class_total]
return sum(class_acc) / len(class_acc)

Common pitfalls

• Confusing the six retrieval configurations (Baseline, Ground-Truth, Oracle-Query RAG, Caption+Question RAG, Multimodal RAG with mmE5/B3) as a single setting; each must be evaluated separately.
• Assuming VLM-as-a-judge replaces the primary metric; it is only used for annotation/reasoning evaluation, while macro-averaged accuracy is the headline metric for multiple-choice answers.
• Overlooking the cross-lingual alignment variable; performance is measured across language mismatches between prompts, context, and model capabilities.

Evidence (verbatim from paper)

For VLMs generations, we use macro-averaged accuracy for all datasets by comparing the multiple choice answer. For annotations we use VLM-as-a-judge using reasoning rubric based since it improves reasoning and more interpretable [3, 4, 19].

Citation

@misc{anugraha2025m4rag,
  title={M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG},
  author={Anugraha et al. (2025)},
  year={2025},
  note={arXiv:2512.05959}
}

• arXiv: 2512.05959