M2rag Multimodal Eval

m2rag-multimodal-eval

MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAG — Wang et al. (2026) (arXiv:2604.24564, 2026)

What this evaluates

Evaluates multimodal retrieval-augmented generation systems across open-domain question answering, image captioning, and fact verification. It measures how effectively a system selects and utilizes retrieved multimodal evidence to improve generation quality and factual accuracy.

Datasets

• M2RAG — total ?; splits: test (-1)

Metrics

• CIDEr (primary) — range: [0, 1]
  • Consensus-based Image Description Evaluation. Measures n-gram overlap between generated and reference captions, weighted by consensus among multiple references.
• BLEU — range: [0, 1]
  • Bilingual Evaluation Understudy. Measures precision of n-gram matches between generated and reference text.
• ROUGE — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation. Measures recall of n-gram overlap between generated and reference text.
• Accuracy — range: [0, 1]
  • Proportion of correctly verified claims or answered questions.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for fact verification tasks.

Input / output format

Input: Multimodal query (text and/or image) with a candidate set of retrieved multimodal documents provided as context.

Output: Natural language generation (answer, caption, or verification statement).

Scoring recipe

def score(predictions, references):
    bleu = corpus_bleu(references, predictions)
    rouge = corpus_rouge(references, predictions)
    cider = corpus_cider(references, predictions)
    acc = mean([p == r for p, r in zip(predictions, references)])
    f1 = f1_score(references, predictions, average='macro')
    return {'BLEU': bleu, 'ROUGE': rouge, 'CIDEr': cider, 'Accuracy': acc, 'F1': f1}

Common pitfalls

• Threshold sensitivity for MEG labeling (τ) significantly impacts training data balance and downstream performance.
• Loss weight (α) between CE and RankNet losses requires careful tuning; extreme values yield sub-optimal results.
• Teacher model choice can introduce bias if not properly distilled, though MEG-RAG shows robustness.

Evidence (verbatim from paper)

Following the M2RAG benchmark, we report BLEU, ROUGE, and CIDEr for MMQA and Image Captioning, and Accuracy and F1 for Fact Verification.

Citation

@misc{wang2026megrag,
  title={MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAG},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.24564}
}

• arXiv: 2604.24564