qhjqhj00 nl-object-retrieval-eval
Natural Language Object Retrieval — Hu et al. (2015) (arXiv:1511.04164, 2015)
What this evaluates
Evaluates a model's ability to ground natural language queries to specific regions within images by scoring candidate bounding boxes. It probes spatial reasoning, contextual understanding, and cross-modal alignment between text and visual features.
Datasets
- ReferIt — total 20000; splits: trainval (10000), test (10000)
- Kitchen — total 606; splits: trainval (300), test (306)
Metrics
P@1(primary) — range: percent- Top-1 precision: the percentage of queries where the highest-scoring candidate region overlaps with the ground truth bounding box by at least 50% IoU.
R@1— range: percent- Recall@1: the percentage of queries where the highest-scoring candidate region overlaps with the ground truth by at least 50% IoU.
R@10— range: percent- Recall@10: the percentage of queries where at least one of the top-10 highest-scoring candidates overlaps with the ground truth by at least 50% IoU.
Input / output format
Input: An image, a set of candidate bounding boxes (either all annotated regions or object proposals), and a natural language query string.
Output: A scalar score for each candidate bounding box, used to rank them. The top-ranked box is returned as the retrieval result.
Scoring recipe
def compute_metrics(scores_list, gt_boxes, iou_thresh=0.5):
correct_top1 = 0
correct_top10 = 0
for scores, gt in zip(scores_list, gt_boxes):
ranked = np.argsort(scores)[::-1]
top1_box = boxes[ranked[0]]
top10_boxes = [boxes[i] for i in ranked[:10]]
if iou(top1_box, gt) >= iou_thresh: correct_top1 += 1
if any(iou(b, gt) >= iou_thresh for b in top10_boxes): correct_top10 += 1
return correct_top1 / len(scores_list), correct_top10 / len(scores_list)
Common pitfalls
- P@1 includes non-informative queries where bag-of-words baselines fail, while P@1-NR excludes them; mixing these up skews comparison.
- IoU threshold is strictly 50% overlap, not the more common 0.75 or 0.5 for detection.
- Kitchen dataset evaluation uses image-level features instead of bounding boxes due to dataset characteristics, requiring model adaptation.
Evidence (verbatim from paper)
Similar to [10], we evaluate with "P@1-NR" corresponding to non-random top-1 precision computed on the those informative results and "P@1" corresponding to top-1 precision on all cases including non-informative results, where random guess is used.
Citation
@misc{hu2015natural,
title={Natural Language Object Retrieval},
author={Hu et al. (2015)},
year={2015},
note={arXiv:1511.04164}
}
- arXiv: 1511.04164