Mig Bench Eval

mig-bench-eval

Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models — You Li et al. (2025) (arXiv:2501.05767, 2025)

What this evaluates

Evaluates a model's ability to perform free-form, multi-image visual grounding by localizing specified objects across multiple input images based on natural language instructions. It probes cross-image reasoning, spatial understanding, and the capacity to follow complex, unstructured queries without relying on chain-of-thought abstractions.

Datasets

• MIG-Bench — total ?; splits: test (-1)

Metrics

• Acc_0.5 (primary) — range: percent
  • Accuracy of object localization where a prediction is considered correct if the Intersection over Union (IoU) between the predicted bounding box and the ground truth bounding box exceeds 0.5.

Input / output format

Input: A set of multiple input images accompanied by a free-form natural language instruction describing the target object(s) to locate.

Output: Bounding box coordinates (typically normalized or pixel-based) corresponding to the predicted location of the target object in the input image(s).

Scoring recipe

correct = 0
total = len(predictions)
for pred_box, gt_box in zip(predictions, ground_truth):
    iou = calculate_iou(pred_box, gt_box)
    if iou > 0.5:
        correct += 1
accuracy = (correct / total) * 100

Common pitfalls

• IoU threshold of 0.5 is standard but may not align with stricter localization requirements in some downstream tasks or datasets.
• Handling queries that reference objects across different images requires correct cross-image alignment, which models often struggle with without explicit grounding training.
• Coordinate format (normalized vs. absolute pixels) must match the evaluation script to avoid IoU calculation errors or off-by-one boundary issues.

Evidence (verbatim from paper)

For the evaluation in our proposed MIG-Bench, we use the conventional metric $ ext{Acc}_{0.5}$ in referring expression comprehension[[20]]. This metric measures the accuracy of object localization, defining a prediction as correct if the Intersection over Union (IoU) with the ground truth is greater than 0.5.

Citation

@misc{li2025migician,
  title={Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models},
  author={You Li et al. (2025)},
  year={2025},
  note={arXiv:2501.05767}
}

• arXiv: 2501.05767