Gres Eval

gres-eval

GRES: Generalized Referring Expression Segmentation — Liu et al. (2023) (arXiv:2306.00968, 2023)

What this evaluates

Evaluates a model's ability to segment arbitrary numbers of target objects (including zero) in an image based on a natural language expression. It probes multi-target localization, no-target rejection, and robustness to complex linguistic structures like counting and compound relations.

Datasets

• gRefCOCO — total 278232; splits: train (-1), val (-1), test (-1)

Metrics

• generalized IoU (gIoU) (primary) — range: [0, 1]
  • Extends mean IoU to all samples, including no-target expressions. For no-target cases, the ground truth is empty, so any non-empty prediction yields an IoU of 0. Computed as the average IoU across all test samples.
• cumulative IoU (cIoU) — range: [0, 1]
  • Standard RES metric computing the mean Intersection over Union across all target instances in the dataset. For multi-target expressions, IoU is computed per target and averaged.
• Precision@X — range: [0, 1]
  • Pixel-wise precision at a given IoU threshold X. The exact threshold value is not specified in the text and follows standard RES conventions.
• No-target-accuracy (N-acc.) — range: percent
  • Percentage of no-target expressions correctly predicted as empty masks.
• Target-accuracy (T-acc.) — range: percent
  • Percentage of target expressions correctly predicted with a non-empty mask above a standard threshold.

Input / output format

Input: An image I and a natural language expression T.

Output: A binary segmentation mask M over the image. For no-target expressions, M must be entirely empty (all negative).

Scoring recipe

def compute_gres_metrics(pred_masks, gold_masks, is_no_target_flags):
    ious = []
    n_acc = 0
    t_acc = 0
    for pred, gt, is_no in zip(pred_masks, gold_masks, is_no_target_flags):
        if is_no:
            ious.append(0.0)
            if not pred.any(): n_acc += 1
        else:
            iou_val = np.sum(pred & gt) / np.sum(pred | gt)
            ious.append(iou_val)
            if iou_val >= 0.5: t_acc += 1
    gIoU = np.mean(ious)
    cIoU = np.mean([np.sum(p & g) / np.sum(p | g) for p, g in zip(pred_masks, gold_masks) if not is_no])
    return {'gIoU': gIoU, 'cIoU': cIoU, 'N-acc.': n_acc/len(pred_masks), 'T-acc.': t_acc/len(pred_masks)}

Common pitfalls

• Models often fail to output an empty mask for no-target expressions, incorrectly segmenting background or irrelevant objects.
• Multi-target expressions require aggregating IoU across multiple instances; failing to correctly sum or average per-instance IoUs leads to inaccurate cIoU/gIoU scores.
• Counting expressions (e.g., 'two people') and ordinal expressions (e.g., 'second person') are easily confused, causing wrong instance selection and degraded precision.

Evidence (verbatim from paper)

Besides the regular RES performance metric cumulative IoU (cIoU) and Precision@X, we further propose a new metric called generalized IoU (gIoU), which extends the mean IoU to all samples including no-target ones. Moreover, No-target performance is also separately evaluated by computing No-target-accuracy (N-acc.) and Target-accuracy (T-acc.).

Citation

@misc{liu2023gres,
  title={GRES: Generalized Referring Expression Segmentation},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.00968}
}

• arXiv: 2306.00968