Refcoco Grounding Eval

refcoco-grounding-eval

DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model — Tao et al. (2025) (arXiv:2512.12633, 2025)

What this evaluates

Evaluates fine-grained visual grounding and spatial reasoning by measuring how accurately a multimodal model can localize regions in images corresponding to given referring expressions under varying linguistic and spatial conditions.

Datasets

• RefCOCO — total ?; splits: val (-1), testA (-1), testB (-1)
• RefCOCO+ — total ?; splits: val (-1), testA (-1), testB (-1)
• RefCOCOg — total ?; splits: val (-1), test (-1)

Metrics

• IoU@50 accuracy (primary) — range: [0, 1]
  • Percentage of correctly localized regions where the Intersection over Union (IoU) between the predicted bounding box and the ground truth box is ≥ 0.5.
• valavg / testAavg / testBavg — range: [0, 1]
  • Average accuracy across IoU thresholds of 50, 75, and 95 for the validation, testA, or testB splits respectively.

Input / output format

Input: An image and a referring expression (text query describing a target object/region).

Output: Bounding box coordinates (typically [x_min, y_min, x_max, y_max] normalized to [0, 1] or in pixel space).

Scoring recipe

def compute_iou(box1, box2):
    x1 = max(box1[0], box2[0])
    y1 = max(box1[1], box2[1])
    x2 = min(box1[2], box2[2])
    y2 = min(box1[3], box2[3])
    inter = max(0, x2 - x1) * max(0, y2 - y1)
    area1 = (box1[2] - box1[0]) * (box1[3] - box1[1])
    area2 = (box2[2] - box2[0]) * (box2[3] - box2[1])
    return inter / (area1 + area2 - inter)

def score(predictions, golds, threshold=0.5):
    correct = sum(1 for pred, gold in zip(predictions, golds) if compute_iou(pred, gold) >= threshold)
    return correct / len(golds)

Common pitfalls

• RefCOCO has two distinct test sets (testA from Flickr30k, testB from COCO); results must be reported separately.
• IoU thresholds of 50, 75, and 95 are standard; reporting only one may misrepresent performance.
• Coordinate format (normalized vs. absolute pixels) must match the ground truth scale to avoid incorrect IoU calculations.

Evidence (verbatim from paper)

Results on referring expression comprehension datasets RefCOCO, RefCOCO+, and RefCOCOg. Incorporating DiG into Qwen3-VL models consistently improves localization accuracy across different IoU thresholds, demonstrating stronger fine-grained spatial grounding and generalization ability.

Citation

@misc{tao2025dig,
  title={DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model},
  author={Tao et al. (2025)},
  year={2025},
  note={arXiv:2512.12633}
}

• arXiv: 2512.12633