Refcocom Eval

refcocom-eval

Towards Unified Referring Expression Segmentation Across Omni-Level Visual Target Granularities — Jing Liu et al. (2025) (arXiv:2504.01954, 2025)

What this evaluates

Evaluates a model's ability to perform referring expression segmentation at both object and part levels. It probes fine-grained cross-modal alignment and pixel-level semantic understanding by requiring precise mask prediction for diverse textual references.

Datasets

• RefCOCOm — total 92000; splits: validation (-1), testA (-1), testB (-1); repo https://github.com/Rubics-Xuan/MRES

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection-over-Union between predicted and ground-truth binary segmentation masks. Computed as the average of |pred ∩ gt| / |pred ∪ gt| across all references.

Input / output format

Input: An image paired with a natural language referring expression.

Output: A binary segmentation mask corresponding to the referenced entity (object or part).

Scoring recipe

ious = []
for pred_mask, gt_mask in zip(predictions, ground_truths):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    if union > 0:
        ious.append(intersection / union)
return np.mean(ious)

Common pitfalls

• Evaluating only on object-level masks ignores the benchmark's core part-level grounding contribution, leading to inflated but misleading scores.
• Models trained exclusively on object-level data often fail to generalize to part-level references without explicit multi-granularity fine-tuning or fine-grained feature exploration.
• Using bounding box metrics instead of pixel-level IoU misaligns with the segmentation task definition and underestimates localization precision.

Evidence (verbatim from paper)

To align with the goal of multi-granularity unification, we use mean Intersection-over-Union (mIoU) as the evaluation metric. Combined with the original object-level annotations, RefCOCOm includes 34k masks and 92k references overall.

Citation

@misc{liu2025refcocom,
  title={Towards Unified Referring Expression Segmentation Across Omni-Level Visual Target Granularities},
  author={Jing Liu et al. (2025)},
  year={2025},
  note={arXiv:2504.01954}
}

• arXiv: 2504.01954