Grefcoco Eval

grefcoco-eval

GREC: Generalized Referring Expression Comprehension — He et al. (2023) (arXiv:2308.16182, 2023)

What this evaluates

Tests a model's ability to ground natural language expressions that may refer to zero, one, or multiple objects in an image. The model must output a corresponding set of bounding boxes rather than a single box, evaluating its capacity for multi-target and no-target referring expression comprehension.

Datasets

• gRefCOCO — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/henghuiding/gRefCOCO

Metrics

• set-matching accuracy (primary) — range: [0, 1]
  • Matches predicted bounding box sets to ground truth sets using an IoU threshold (typically 0.5). A prediction is correct if every predicted box matches a ground truth box within the threshold and set sizes align. Accuracy is the percentage of correctly matched expressions across the dataset.

Input / output format

Input: An image and a natural language referring expression.

Output: A set of bounding boxes $B={b_i}$, where each $b_i$ is a predicted bounding box. The set size ranges from 0 (no-target expression) to multiple (multi-target expression).

Scoring recipe

def compute_set_matching_accuracy(predictions, ground_truth, iou_threshold=0.5):
    correct = 0
    for pred_boxes, gt_boxes in zip(predictions, ground_truth):
        if len(pred_boxes) != len(gt_boxes):
            continue
        matched = 0
        used_gt = set()
        for p in pred_boxes:
            best_iou = 0
            best_idx = -1
            for idx, g in enumerate(gt_boxes):
                if idx not in used_gt:
                    iou = compute_iou(p, g)
                    if iou > best_iou:
                        best_iou = iou
                        best_idx = idx
            if best_iou >= iou_threshold:
                matched += 1
                used_gt.add(best_idx)
        if matched == len(gt_boxes):
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Models trained on classic REC datasets assume exactly one target and will incorrectly output a single box for multi-target or no-target expressions, leading to artificially low scores if not handled.
• Evaluating with single-box matching metrics instead of set-based matching leads to unfairly low scores for models that correctly identify multiple objects but output them in a different order or with slight coordinate variations.

Evidence (verbatim from paper)

In contrast to classic REC, which outputs a single bounding box for a referring expression, GREC aims to output a set of bounding boxes $B={b_{i}}$, where each $b_{i}$ corresponds to an object among all the target objects referred by the given expression. The number of bounding boxes can range from 0 to multiple, depending on the given expression.

Citation

@misc{he2023grec,
  title={GREC: Generalized Referring Expression Comprehension},
  author={He et al. (2023)},
  year={2023},
  note={arXiv:2308.16182}
}

• arXiv: 2308.16182