Groundingme Eval

groundingme-eval

GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation — Li et al. (2025) (arXiv:2512.17495, 2025)

What this evaluates

Evaluates multimodal large language models' (MLLMs) visual grounding capabilities across four dimensions: discriminative object distinction, spatial relational understanding, handling occlusion/size constraints, and the ability to reject ungroundable queries. It measures how well models can localize objects in images and whether they hallucinate or correctly refuse impossible requests.

Datasets

• GroundingME — total ?; splits: test (-1)

Metrics

• Accuracy@0.5 (primary) — range: percent
  • The percentage of test samples where the Intersection over Union (IoU) between the predicted bounding box and the ground-truth bounding box exceeds 0.5.

Input / output format

Input: An input image paired with a textual description or query, formatted using a unified prompt template that specifies viewpoint references, allowed output counts, and strict formatting constraints.

Output: Bounding box coordinates for the target object(s), strictly following the template's format constraints and output count limits.

Scoring recipe

def accuracy_at_0_5(predictions, ground_truths):
    correct = 0
    for pred, gt in zip(predictions, ground_truths):
        if calculate_iou(pred, gt) > 0.5:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Models often hallucinate objects for invalid queries instead of correctly rejecting them, severely depressing rejection scores.
• Reporting only the overall average accuracy masks significant performance stratification across subcategories (e.g., high discriminative scores vs. near-zero rejection scores).
• The default evaluation protocol uses greedy decoding (temperature=0), which may not reflect each model's peak performance.

Evidence (verbatim from paper)

For the evaluation metric, we adopt the widely-used Accuracy@0.5, which represents the proportion of total samples where the Intersection over Union (IoU) between the ground-truth and predicted bounding box exceeds 0.5.

Citation

@misc{li2025groundingme,
  title={GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2512.17495}
}

• arXiv: 2512.17495