qhjqhj00 visual-relationship-detection-eval
Visual Semantic Information Pursuit: A Survey — Daqi Liu et al. (2019) (arXiv:1903.05434, 2019)
What this evaluates
Probes a model's ability to identify and classify interactions between pairs of objects in an image (subject-predicate-object triples). It focuses on capturing relational semantics beyond isolated object detection.
Datasets
- Visual Relationship Dataset — total 5000; splits: test (5000)
- Visual Genome — total 108077; splits: test (108077)
Metrics
recall@50(primary) — range: [0, 1]- Fraction of ground-truth relationships correctly predicted within the top 50 highest-confidence relationship predictions per image.
recall@100— range: [0, 1]- Fraction of ground-truth relationships correctly predicted within the top 100 highest-confidence relationship predictions per image.
Input / output format
Input: RGB image with detected object bounding boxes and class labels.
Output: Ranked list of predicted subject-predicate-object triples with confidence scores.
Scoring recipe
def compute_recall_at_x(preds, gts, x):
top_preds = sorted(preds, key=lambda p: p.confidence, reverse=True)[:x]
matches = sum(1 for gt in gts if any(match(triple(gt), triple(p)) for p in top_preds))
return matches / len(gts) if gts else 0
Common pitfalls
- Recall@x is used instead of mAP because exhaustive ground-truth annotation of all possible relationships is infeasible; mAP would unfairly penalize correct but unannotated relationships.
- Predictions are often filtered to frequent predicates/categories; evaluating on full vocabulary inflates error rates.
- Top-x ranking is per-image, not global; rare relationships suffer heavily.
Evidence (verbatim from paper)
The current visual relationship detection methods often use two evaluation metrics: recall@50 and recall@100. Here, recall@x [89] represents the fraction of times the correct relationship is predicted in the top x confident relationship predictions. The reason why we use recall@x instead of widely applied mean average precision (mAP) metric is because mAP is a pessimistic evaluation metric, meaning we can not exhaustively annotate all possible relationships in an image. Even if the prediction is correct, mAP still would penalize the prediction if we do not have that particular ground truth annotation.
Citation
@misc{liu2019visualsemantic,
title={Visual Semantic Information Pursuit: A Survey},
author={Daqi Liu et al. (2019)},
year={2019},
note={arXiv:1903.05434}
}
- arXiv: 1903.05434