Referring Segmentation Eval

referring-segmentation-eval

Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos — Haobo Yuan et al. (arXiv:2501.04001, 2025)

What this evaluates

Evaluates a model's ability to perform dense grounded understanding by localizing and segmenting specific objects in images and videos based on natural language instructions or referring expressions.

Datasets

• Ref-SAV — total 37000; splits: val (37000)
• RefCOCO — total 17000; splits: test (-1)
• RefCOCO+ — total 17000; splits: test (-1)
• RefCOCOg — total 22000; splits: test (-1)
• MeVIS — total 600; splits: test (-1)
• Ref-YTVOS — total 3500; splits: test (-1)
• ReVOS — total 1700; splits: test (-1)

Metrics

• cIoU (primary) — range: [0, 1]
  • Conditional Intersection over Union between the predicted segmentation mask and the ground-truth mask, conditioned on the referring expression.
• J&F — range: [0, 1]
  • Joint and F-measure, computed as the average of frame-wise IoU (J) and boundary F-score (F) across all video frames.

Input / output format

Input: Image or video frames accompanied by a text prompt (referring expression or instruction).

Output: A segmentation mask (binary or probability map) for the referred object, generated by decoding the '[SEG]' token hidden state through SAM2's decoder.

Scoring recipe

def compute_cIoU(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    return intersection / union if union > 0 else 0.0

def compute_JF(pred_masks, gt_masks):
    J = np.mean([compute_cIoU(p, g) for p, g in zip(pred_masks, gt_masks)])
    F = compute_boundary_F1(pred_masks, gt_masks)
    return (J + F) / 2

Common pitfalls

• cIoU conditions on the referring expression; evaluating without aligning the mask to the specific object mentioned in the prompt yields invalid scores.
• J&F requires strict frame-by-frame evaluation for videos; averaging masks across frames or ignoring temporal consistency will produce incorrect results.
• The mask must be decoded specifically from the '[SEG]' token's hidden state via SAM2's decoder; using alternative decoding heads will not match reported baselines.

Evidence (verbatim from paper)

For image referring segmentation, we adopt cIoU. For referring video object segmentation, we adopt J&F. ... the segmentation mask is obtained by decoding the hidden state of the “[SEG]” token through SAM2’s decoder.

Citation

@misc{yuan2025sa2va,
  title={Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos},
  author={Haobo Yuan et al.},
  year={2025},
  note={arXiv:2501.04001}
}

• arXiv: 2501.04001