# Nl Object Retrieval Eval

> Evaluates a model's ability to ground natural language queries to specific regions within images by scoring candidate bounding boxes. It probes spatial reasoning, contextual understanding, and cross-modal alignment between text and visual features. Use when the user wants to benchmark on ReferIt, Kitchen, or asks about evaluating this task. Reports P@1.

- Skill: `qhjqhj00/nl-object-retrieval-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/nl-object-retrieval-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/nl-object-retrieval-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/nl-object-retrieval-eval

---


# nl-object-retrieval-eval

> Natural Language Object Retrieval — Hu et al. (2015) (arXiv:1511.04164, 2015)

## What this evaluates

Evaluates a model's ability to ground natural language queries to specific regions within images by scoring candidate bounding boxes. It probes spatial reasoning, contextual understanding, and cross-modal alignment between text and visual features.

## Datasets

- **ReferIt** — total 20000; splits: trainval (10000), test (10000)
- **Kitchen** — total 606; splits: trainval (300), test (306)

## Metrics

- `P@1` **(primary)** — range: percent
  - Top-1 precision: the percentage of queries where the highest-scoring candidate region overlaps with the ground truth bounding box by at least 50% IoU.
- `R@1` — range: percent
  - Recall@1: the percentage of queries where the highest-scoring candidate region overlaps with the ground truth by at least 50% IoU.
- `R@10` — range: percent
  - Recall@10: the percentage of queries where at least one of the top-10 highest-scoring candidates overlaps with the ground truth by at least 50% IoU.

## Input / output format

**Input**: An image, a set of candidate bounding boxes (either all annotated regions or object proposals), and a natural language query string.

**Output**: A scalar score for each candidate bounding box, used to rank them. The top-ranked box is returned as the retrieval result.

## Scoring recipe

```python
def compute_metrics(scores_list, gt_boxes, iou_thresh=0.5):
    correct_top1 = 0
    correct_top10 = 0
    for scores, gt in zip(scores_list, gt_boxes):
        ranked = np.argsort(scores)[::-1]
        top1_box = boxes[ranked[0]]
        top10_boxes = [boxes[i] for i in ranked[:10]]
        if iou(top1_box, gt) >= iou_thresh: correct_top1 += 1
        if any(iou(b, gt) >= iou_thresh for b in top10_boxes): correct_top10 += 1
    return correct_top1 / len(scores_list), correct_top10 / len(scores_list)
```

## Common pitfalls

- P@1 includes non-informative queries where bag-of-words baselines fail, while P@1-NR excludes them; mixing these up skews comparison.
- IoU threshold is strictly 50% overlap, not the more common 0.75 or 0.5 for detection.
- Kitchen dataset evaluation uses image-level features instead of bounding boxes due to dataset characteristics, requiring model adaptation.

## Evidence (verbatim from paper)

> Similar to [10], we evaluate with "P@1-NR" corresponding to non-random top-1 precision computed on the those informative results and "P@1" corresponding to top-1 precision on all cases including non-informative results, where random guess is used.

## Citation

```bibtex
@misc{hu2015natural,
  title={Natural Language Object Retrieval},
  author={Hu et al. (2015)},
  year={2015},
  note={arXiv:1511.04164}
}
```

- arXiv: 1511.04164