Vgphrasecut Eval

vgphrasecut-eval

PhraseCut: Language-based Image Segmentation in the Wild — Chenyun Wu et al. (2020) (arXiv:2008.01187, 2020)

What this evaluates

This benchmark evaluates language-based image segmentation by requiring models to ground natural language phrases into precise image regions. It probes a model's ability to handle long-tail categories, attributes, relationships, and varying object sizes in open-vocabulary settings.

Datasets

• VGPhraseCut — total ?; splits: test (-1)

Metrics

• mean-IoU (primary) — range: [0, 1]
  • Average Intersection over Union across all test instances. IoU is computed as the area of overlap between the predicted and ground-truth segmentation masks divided by the area of their union.
• cum-IoU — range: [0, 1]
  • Cumulative Intersection over Union, calculated as the total intersection area across all instances divided by the total union area. Dominated by large targets.
• Pr@0.5 — range: [0, 1]
  • Precision at IoU threshold 0.5, measuring the fraction of predictions with IoU ≥ 0.5.

Input / output format

Input: An input image and a natural language referring expression (phrase) specifying a target object or region.

Output: A binary segmentation mask (or region proposal) corresponding to the target described by the phrase.

Scoring recipe

def compute_mean_iou(predictions, ground_truths):
    ious = []
    for pred, gt in zip(predictions, ground_truths):
        intersection = np.logical_and(pred, gt).sum()
        union = np.logical_or(pred, gt).sum()
        ious.append(intersection / union if union > 0 else 0.0)
    return np.mean(ious)

Common pitfalls

• RMI achieves high cum-IoU but low mean-IoU because it performs well on large targets but fails on small ones, skewing cumulative metrics.
• MattNet's evaluation relies on ground-truth instance boxes for context; using predicted detections instead causes a severe performance drop, highlighting dataset sparsity issues.
• Performance varies drastically across category frequency subsets (top 100 vs 500+), so reporting only overall mean-IoU masks long-tail failures.

Evidence (verbatim from paper)

RMI obtains high cum-IoU but low mean-IoU scores because it handles large targets well but fails on small ones (see Table 4 “small/mid/large” subsets). cum-IoU is dominated by large targets while our dataset many small targets: 20.2% of our data has the target region smaller than 2% of the image area, while the smallest target in RefCOCO is 2.4% of the image.

Citation

@misc{wu2020phrasecut,
  title={PhraseCut: Language-based Image Segmentation in the Wild},
  author={Chenyun Wu et al. (2020)},
  year={2020},
  note={arXiv:2008.01187}
}

• arXiv: 2008.01187