Grounder Eval

grounder-eval

Grounding of Textual Phrases in Images by Reconstruction — Rohrbach et al. (2015) (arXiv:1511.03745, 2015)

What this evaluates

This benchmark evaluates a model's ability to localize arbitrary natural language phrases within images. It probes phrase grounding capabilities by requiring the model to attend to relevant image regions and select a bounding box that matches the textual description, without relying on explicit bounding box supervision during training.

Datasets

• Flickr 30k Entities — total 31000; splits: train (29783), val (1000), test (1000)
• ReferItGame — total 20000; splits: train (9000), val (1000), test (10000)

Metrics

• Accuracy (primary) — range: percent
  • Ratio of phrases for which the attended bounding box overlaps with the ground-truth box by more than 0.5 IOU. For phrases corresponding to multiple boxes, the union of boxes is used as ground truth.

Input / output format

Input: An image and a natural language phrase describing a target region within that image.

Output: A single bounding box selected from a set of pre-computed proposals (or attention weights over those proposals).

Scoring recipe

def compute_accuracy(predictions, gold_boxes):
    correct = 0
    for pred_box, gold_box in zip(predictions, gold_boxes):
        iou = intersection_over_union(pred_box, gold_box)
        if iou > 0.5:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Multiple boxes per phrase: the union of all corresponding ground-truth boxes is used as the single target.
• Proposal generation differs by dataset: Selective Search is used for Flickr 30k Entities, while Edge Boxes are used for ReferItGame.
• Semi-supervised training only uses phrase-box pairs where a proposal overlaps the ground truth by >0.5 IOU.
• Visual features vary: VGG-CLS (ImageNet), VGG-DET (PASCAL fine-tuned), and VGG+SPAT (ReferItGame specific spatial features).

Evidence (verbatim from paper)

At test time we compute the accuracy as the ratio of phrases for which the attended box overlaps with the ground-truth box by more than 0.5 IOU. For consistency with [[35]], in such cases we consider the union of the boxes as ground truth.

Citation

@misc{rohrbach2015grounding,
  title={Grounding of Textual Phrases in Images by Reconstruction},
  author={Rohrbach et al. (2015)},
  year={2015},
  note={arXiv:1511.03745}
}

• arXiv: 1511.03745