Mrag Bench Eval

mrag-bench-eval

MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models — Hu et al. (2024) (arXiv:2410.08182, 2024)

What this evaluates

Evaluates large vision-language models' ability to leverage retrieved visual knowledge versus textual knowledge across perspective and transformative change scenarios. Probes robustness to noisy retrieved images and measures how effectively models utilize visually augmented information compared to human baselines.

Datasets

• MRAG-Bench — total 1353; splits: test (1353)

Metrics

• accuracy (primary) — range: percent
  • Standard multiple-choice question accuracy. Calculated as the number of correctly answered questions divided by the total number of questions, expressed as a percentage.

Input / output format

Input: Multi-image inputs paired with multiple-choice questions. Models are evaluated under three conditions: no additional knowledge, ground-truth image knowledge, and retrieved image knowledge from a multimodal retriever.

Output: A single multiple-choice answer. If pre-defined automatic extraction rules fail, GPT-3.5-turbo is used to extract the answer from the model's generation.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if pred == gold)
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Automatic answer extraction rules may fail on some generations, requiring a fallback to GPT-3.5-turbo for reliable scoring.
• Open-source models often show performance degradation when provided with retrieved images compared to no RAG, due to difficulty filtering noisy or misleading visual examples.
• Performance varies significantly across the 9 fine-grained scenarios (e.g., perspective vs. transformative, incomplete/biological), so reporting only aggregate scores masks critical capability gaps.

Evidence (verbatim from paper)

We follow standard MCQA evaluation setup and employ accuracy score as our metric. We adopt default generation hyper-parameters selected by each model. Following Lu et al. ([2024b]), we employ GPT-3.5-turbo to extract the multiple choice answer in rare cases where our pre-defined automatic extraction rules failed.

Citation

@misc{hu2024mragbench,
  title={MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models},
  author={Hu et al. (2024)},
  year={2024},
  note={arXiv:2410.08182}
}

• arXiv: 2410.08182