Referring Expression Comprehension Eval

referring-expression-comprehension-eval

Modeling Context Between Objects for Referring Expression Understanding — Nagaraja et al. (2016) (arXiv:1608.00525, 2016)

What this evaluates

Probes a model's ability to ground referring expressions in images by understanding spatial and relational context between objects. It evaluates weakly supervised region proposal scoring and context pooling mechanisms without requiring explicit bounding box annotations for context regions.

Datasets

• Google RefExp — total ?; splits: train (85408), val (9602)
• UNC RefExp — total ?; splits: train (120624), val (10834), testA (5657), testB (5095)

Metrics

• Precision@1 (primary) — range: percent
  • Averaged over all referring expressions: (1/N) * Σ [IoU(pred_box, gt_box) > 0.5], where a prediction is a true positive if the Intersection over Union exceeds 0.5.

Input / output format

Input: An image, a referring expression (text), and a set of region proposals (bounding boxes) with extracted CNN features.

Output: A single predicted bounding box (x_min, y_min, x_max, y_max) corresponding to the referred object in the image.

Scoring recipe

def compute_precision_at_1(predictions, ground_truths):
    correct = 0
    for pred_box, gt_box in zip(predictions, ground_truths):
        iou = intersection_over_union(pred_box, gt_box)
        if iou > 0.5:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• The original Google RefExp partition had overlapping images across train/val/test splits, causing data leakage; the authors explicitly created a non-overlapping partition to fix this.
• UNC RefExp TestA (person-centric) and TestB (object-centric) splits differ significantly in expression terseness, causing context modeling to help less on TestA.
• Testing uses noisy-or pooling over up to 10 sampled context regions, making scores sensitive to the quality of region proposals (MCG vs. ground truth).

Evidence (verbatim from paper)

The evaluation is performed by measuring the Intersection over Union (IoU) ratio between a groundtruth box and the top predicted box for a referring expression. If the IoU >0.5, the prediction is considered a true positive and this is the Precision@1 score. The scores are then averaged over all referring expressions.

Citation

@misc{nagaraja2016modeling,
  title={Modeling Context Between Objects for Referring Expression Understanding},
  author={Nagaraja et al. (2016)},
  year={2016},
  note={arXiv:1608.00525}
}

• arXiv: 1608.00525