Referit Segmentation Eval

referit-segmentation-eval

Segmentation from Natural Language Expressions — Hu et al. (2016) (arXiv:1603.06180, 2016)

What this evaluates

This benchmark evaluates a model's ability to perform pixel-level image segmentation conditioned on natural language expressions. It probes spatial reasoning, attribute grounding, and fine-grained visual-linguistic alignment by requiring the model to segment specific objects or amorphous regions described in text.

Datasets

• ReferIt — total 20000; splits: trainval (10000), test (10000)

Metrics

• overall IoU — range: percent
  • Total intersection area divided by total union area, where both intersection and union areas are accumulated over all test samples.
• prec@0.5 (primary) — range: percent
  • Percentage of test samples where the IoU between predicted and ground-truth segmentation is ≥ 0.5. The paper also reports prec@0.6 through prec@0.9 using the same definition.

Input / output format

Input: An RGB image and a natural language expression (referential description) targeting a specific region in the image.

Output: A pixelwise segmentation mask (binary or probability map) aligned with the input image dimensions.

Scoring recipe

total_intersection = 0
total_union = 0
correct_counts = {0.5: 0, 0.6: 0, 0.7: 0, 0.8: 0, 0.9: 0}
for pred, gold in zip(predictions, golds):
    intersection = np.sum((pred == 1) & (gold == 1))
    union = np.sum((pred == 1) | (gold == 1))
    total_intersection += intersection
    total_union += union
    iou = intersection / union if union > 0 else 0.0
    for t in [0.5, 0.6, 0.7, 0.8, 0.9]:
        if iou >= t:
            correct_counts[t] += 1
overall_iou = total_intersection / total_union
prec_at_t = {t: correct_counts[t] / len(predictions) for t in [0.5, 0.6, 0.7, 0.8, 0.9]}

Common pitfalls

• Overall IoU heavily favors large background regions (e.g., sky, ground), making it a misleading metric for fine-grained referential segmentation; the authors explicitly recommend the precision metric instead.
• The dataset contains both discrete 'object' regions and amorphous 'stuff' regions (e.g., sky, river), which require different segmentation behaviors but are evaluated under the same protocol.
• Images are resized and padded to 512x512 during training/inference, so evaluation must account for aspect-ratio preservation and padding artifacts when computing IoU.

Evidence (verbatim from paper)

The following two metrics are used for evaluation: the overall intersection-over-union (overall IoU) metric and the precision metric. The overall IoU is the total intersection area divided by the total union area, where both intersection area and union area are accumulated over all test samples (each test sample is an image and a referential expression). Although the overall IoU metric is the standard metric used in PASCAL VOC segmentation [11], our evaluation is slightly different as we would like to measure how accurate the model can segment the foreground region described by the input expression against the background, and the overall IoU metric favors large regions like sky and ground. So we also evaluate with the precision metric at 5 different IoU thresholds from easy to hard: 0.5, 0.6, 0.7, 0.8, 0.9. The precision metric is the percentage of test samples where the IoU between prediction and ground-truth passes the threshold.

Citation

@misc{hu2016segmentation,
  title={Segmentation from Natural Language Expressions},
  author={Hu et al. (2016)},
  year={2016},
  note={arXiv:1603.06180}
}

• arXiv: 1603.06180