# Visual Relationship Detection Eval

> Probes a model's ability to identify and classify interactions between pairs of objects in an image (subject-predicate-object triples). It focuses on capturing relational semantics beyond isolated object detection. Use when the user wants to benchmark on Visual Relationship Dataset, Visual Genome, or asks about evaluating this task. Reports recall@50.

- Skill: `qhjqhj00/visual-relationship-detection-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/visual-relationship-detection-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/visual-relationship-detection-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/visual-relationship-detection-eval

---


# visual-relationship-detection-eval

> Visual Semantic Information Pursuit: A Survey — Daqi Liu et al. (2019) (arXiv:1903.05434, 2019)

## What this evaluates

Probes a model's ability to identify and classify interactions between pairs of objects in an image (subject-predicate-object triples). It focuses on capturing relational semantics beyond isolated object detection.

## Datasets

- **Visual Relationship Dataset** — total 5000; splits: test (5000)
- **Visual Genome** — total 108077; splits: test (108077)

## Metrics

- `recall@50` **(primary)** — range: [0, 1]
  - Fraction of ground-truth relationships correctly predicted within the top 50 highest-confidence relationship predictions per image.
- `recall@100` — range: [0, 1]
  - Fraction of ground-truth relationships correctly predicted within the top 100 highest-confidence relationship predictions per image.

## Input / output format

**Input**: RGB image with detected object bounding boxes and class labels.

**Output**: Ranked list of predicted subject-predicate-object triples with confidence scores.

## Scoring recipe

```python
def compute_recall_at_x(preds, gts, x):
    top_preds = sorted(preds, key=lambda p: p.confidence, reverse=True)[:x]
    matches = sum(1 for gt in gts if any(match(triple(gt), triple(p)) for p in top_preds))
    return matches / len(gts) if gts else 0
```

## Common pitfalls

- Recall@x is used instead of mAP because exhaustive ground-truth annotation of all possible relationships is infeasible; mAP would unfairly penalize correct but unannotated relationships.
- Predictions are often filtered to frequent predicates/categories; evaluating on full vocabulary inflates error rates.
- Top-x ranking is per-image, not global; rare relationships suffer heavily.

## Evidence (verbatim from paper)

> The current visual relationship detection methods often use two evaluation metrics: recall@50 and recall@100. Here, recall@x [89] represents the fraction of times the correct relationship is predicted in the top x confident relationship predictions. The reason why we use recall@x instead of widely applied mean average precision (mAP) metric is because mAP is a pessimistic evaluation metric, meaning we can not exhaustively annotate all possible relationships in an image. Even if the prediction is correct, mAP still would penalize the prediction if we do not have that particular ground truth annotation.

## Citation

```bibtex
@misc{liu2019visualsemantic,
  title={Visual Semantic Information Pursuit: A Survey},
  author={Daqi Liu et al. (2019)},
  year={2019},
  note={arXiv:1903.05434}
}
```

- arXiv: 1903.05434