qhjqhj00 mir-eval
From Easy to Hard: The MIR Benchmark for Progressive Interleaved Multi-Image Reasoning — Du et al. (2025) (arXiv:2509.17040, 2025)
What this evaluates
Evaluates multimodal large language models on progressive, interleaved multi-image reasoning tasks. It probes the model's ability to perform structured, step-by-step reasoning across multiple images, including text-to-region alignment, cross-image relationship modeling, and analytical inference.
Datasets
- MIR — total 22257; splits: spatial (-1), Sequential (-1), Analytical (-1); repo https://github.com/Shelly-coder239/MIRBench
Metrics
accuracy(primary) — range: percent- Percentage of correctly matched answers. Due to the long, structured reasoning outputs, rule-based matching is unreliable; the authors uniformly employ GPT to match model outputs with ground truth answers.
Input / output format
Input: Interleaved multi-image sequences paired with a question requiring step-by-step reasoning.
Output: Structured reasoning process consisting of five explicit steps: Summary, Caption, Text2Region, Region2Region, and Conclusion, followed by the final answer.
Scoring recipe
def compute_accuracy(predictions, gold):
matches = 0
for pred, gold_ans in zip(predictions, gold):
# GPT is used to determine if the prediction matches the ground truth
if gpt_match_answer(pred, gold_ans):
matches += 1
return (matches / len(predictions)) * 100
Common pitfalls
- Long model outputs often fail to align with ground truth using standard rule-based answer matching, necessitating GPT-based evaluation.
- Sequential tasks use images extracted from videos, which may not accurately represent the full video context, leading to artificially low performance on this sub-task.
Evidence (verbatim from paper)
During the evaluation phase, since lmms-eval relies on rule-based answer matching, long outputs often struggle to align with the correct answers. Therefore, we uniformly employ GPT to match model outputs with the ground truth. This evaluation scheme ensures the reliability and comprehensiveness of the experimental results.
Citation
@misc{du2025mir,
title={From Easy to Hard: The MIR Benchmark for Progressive Interleaved Multi-Image Reasoning},
author={Du et al. (2025)},
year={2025},
note={arXiv:2509.17040}
}
- arXiv: 2509.17040