Referring Image Segmentation Eval

referring-image-segmentation-eval

Latent Expression Generation for Referring Image Segmentation and Grounding — Yu et al. (2025) (arXiv:2508.05123, 2025)

What this evaluates

Evaluates visual grounding capabilities by measuring how well a model segments or localizes objects in images based on natural language descriptions. It probes the model's ability to handle ambiguous references, diverse textual forms, and generalized referring expressions without task-specific decoders.

Datasets

• RefCOCO — total ?; splits: val (-1), testA (-1), testB (-1)
• RefCOCO+ — total ?; splits: val (-1), testA (-1), testB (-1)
• RefCOCOg — total ?; splits: val (-1), test (-1)
• gRefCOCO — total ?; splits: test (-1)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union; computed as the average IoU across all samples.
• oIoU — range: [0, 1]
  • Overall IoU; computed by dividing the total intersection by the total union over all samples.
• Prec@X — range: [0, 1]
  • Percentage of samples where the predicted mask IoU exceeds a given threshold X (e.g., 0.5, 0.7, 0.9).
• N-acc — range: [0, 1]
  • No-target accuracy for GRES; defined as TP / (TP + FN), where TP correctly predicts no target and FN incorrectly predicts a target.

Input / output format

Input: RGB image (resized to 480×480) paired with a natural language referring expression.

Output: Binary segmentation mask (generated by thresholding model output at 0.35).

Scoring recipe

def compute_metrics(pred_masks, gt_masks, no_target_idx=None):
    ious = [iou(p, g) for p, g in zip(pred_masks, gt_masks)]
    mIoU = sum(ious) / len(ious)
    oIoU = sum(inter(p, g) for p, g in zip(pred_masks, gt_masks)) / sum(union(p, g) for p, g in zip(pred_masks, gt_masks))
    prec_50 = sum(1 for i in ious if i > 0.5) / len(ious)
    prec_70 = sum(1 for i in ious if i > 0.7) / len(ious)
    prec_90 = sum(1 for i in ious if i > 0.9) / len(ious)
    n_acc = 0.0
    if no_target_idx is not None:
        tp = sum(1 for i in no_target_idx if ious[i] == 0)
        fn = sum(1 for i in no_target_idx if ious[i] > 0)
        n_acc = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return mIoU, oIoU, prec_50, prec_70, prec_90, n_acc

Common pitfalls

• Confusing mIoU (per-sample average) with oIoU (global intersection over global union), which yield different values on imbalanced datasets.
• N-acc inverts standard detection logic: a True Positive means the model correctly predicts the absence of a target (IoU=0), not a successful localization.
• Prec@X thresholds are applied independently per sample, not as a global ranking cutoff.

Evidence (verbatim from paper)

For RIS, we use the standard metrics: mIoU, oIoU, and prec@X; mIoU (mean IoU) is the average IoU across all samples, while oIoU (overall IoU) divides the total intersection by the total union over all samples. Prec@X is the percentage of samples with IoU above threshold X. For REC, we employ Prec@50 (Acc.), which is an accuracy only when IoU is over 0.5. For GRES, in addition to mIoU and oIoU, we also compute N-acc (No-target accuracy) to evaluate the accuracy on no-target samples. N-acc is defined as $\frac{TP}{TP+FN}$, where a true positive (TP) indicates correctly predicting no target, otherwise a false negative (FN).

Citation

@misc{yu2025latent,
  title={Latent Expression Generation for Referring Image Segmentation and Grounding},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2508.05123}
}

• arXiv: 2508.05123