Ref Adv Eval

ref-adv-eval

Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks — Dong et al. (2026) (arXiv:2602.23898, 2026)

What this evaluates

Probes multimodal large language models' ability to perform visual grounding and complex textual reasoning under challenging conditions. It specifically tests whether models rely on shortcut cues or genuinely comprehend referring expressions when faced with linguistically nontrivial descriptions and hard distractors.

Datasets

• Ref-Adv — total ?; splits: test (-1)

Metrics

• Acc0.5 (primary) — range: percent
  • Fraction of instances where the Intersection over Union (IoU) between the predicted bounding box and ground truth exceeds 0.5.
• Acc0.75 — range: percent
  • Fraction of instances where IoU between predicted and ground truth bounding boxes exceeds 0.75.
• Acc0.9 — range: percent
  • Fraction of instances where IoU between predicted and ground truth bounding boxes exceeds 0.9.
• mAcc — range: percent
  • Mean accuracy computed across the IoU thresholds 0.5, 0.75, and 0.9.

Input / output format

Input: An image and a referring expression (text prompt). For GPT-4o and Claude-3.5, Set-of-Marks (SoM) overlays with numbered candidate objects are provided instead of raw images to compensate for limited grounding capabilities.

Output: A bounding box coordinate representation (absolute or normalized depending on the model). CogVLM-Grounding requires a strict prompt format to output boxes.

Scoring recipe

def compute_metrics(predictions, ground_truths, thresholds=[0.5, 0.75, 0.9]):
    correct = {t: 0 for t in thresholds}
    for pred_box, gt_box in zip(predictions, ground_truths):
        iou = intersection_area(pred_box, gt_box) / union_area(pred_box, gt_box)
        for t in thresholds:
            if iou >= t:
                correct[t] += 1
    total = len(predictions)
    acc = {t: (correct[t] / total) * 100 for t in thresholds}
    macc = sum(acc.values()) / len(thresholds)
    return acc, macc

Common pitfalls

• Using Set-of-Marks (SoM) for all models instead of only for GPT-4o and Claude-3.5, which artificially inflates performance by offloading grounding to an external segmenter.
• Evaluating only on subsets with few distractors; performance drops significantly when distractor count reaches ≥7, masking true reasoning limitations.
• Assuming Chain-of-Thought (CoT) prompting universally improves results; while beneficial for Ref-Adv's reasoning demands, CoT can introduce verbosity and harm performance on simpler grounding benchmarks.

Evidence (verbatim from paper)

Accuracy serves as a widely adopted metric for evaluating existing REC models. A referring expression instance is deemed successfully grounded when the Intersection over Union (IoU) between the predicted bounding box and the ground truth annotation surpasses 0.5. This conventional evaluation metric is designated as Acc0.5. Here, we implement multiple evaluation protocols, i.e., Accuracy computed under different IoU thresholds such as Acc0.5, Acc0.75, Acc0.9, and mean Accuracy (mAcc) across different IoU criteria, to thoroughly evaluate the precision and robustness.

Citation

@misc{dong2026refadv,
  title={Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks},
  author={Dong et al. (2026)},
  year={2026},
  note={arXiv:2602.23898}
}

• arXiv: 2602.23898