qhjqhj00 scene-graph-generation-eval
Unbiased Scene Graph Generation from Biased Training — Tang et al. (2020) (arXiv:2002.11949, 2020)
What this evaluates
Evaluates scene graph generation models on predicting subject-predicate-object triplets while mitigating long-tailed training biases. It probes zero-shot generalization and graph-level semantic coherence through sentence-to-graph retrieval.
Datasets
- Visual Genome (VG) — total 108000; splits: train (-1), val (5000), test (-1); repo https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch
- MS-COCO Caption (VG Overlap) — total 41859; splits: train (35859), test-1k (1000), test-5k (5000)
Metrics
mR@K(primary) — range: [0, 100] percent- Mean Recall@K. Computes Recall@K for each predicate category separately, then averages across all 50 predicate categories. mR@K = (1/|P|) * Σ_p Recall@K(p).
R@K / Med— range: [0, 100] percent | rank- Recall@K for image retrieval and Median Ranking (Med) of retrieved results. R@K is the fraction of queries where the correct image appears in the top K retrieved results. Med is the median rank of the ground truth image in the ranked list.
Input / output format
Input: Images with ground-truth bounding boxes and labels (PredCls), images with ground-truth boxes but no labels (SGCls), or raw images (SGDet). For S2GR, image captions paired with detected scene graphs.
Output: Predicted subject-predicate-object triplets per image, ranked by confidence score. For S2GR, a ranked list of retrieved images.
Scoring recipe
def compute_mR_at_K(predictions, ground_truth, K, predicates):
recalls = []
for p in predicates:
gt_p = [g for g in ground_truth if g.predicate == p]
pred_p = [pred for pred in predictions if pred.predicate == p]
correct = sum(1 for g in gt_p if g.predicate in [pred.label for pred in pred_p[:K]])
recalls.append(correct / len(gt_p) if gt_p else 0)
return (sum(recalls) / len(predicates)) * 100
Common pitfalls
- Standard Recall@K heavily favors head predicates and masks tail predicate failures; mR@K must be used to fairly evaluate long-tailed bias mitigation.
- Zero-shot evaluation strictly excludes any triplet observed during training, requiring careful dataset splitting and filtering.
- S2GR requires filtering low-confidence RoIs and removing background predicates to align heterogeneous image and text graphs before retrieval.
Evidence (verbatim from paper)
The conventional metric of RR is Recall@K (R@K), which was abandoned in this paper due to the reporting bias[[36]]. As illustrated in Figure3, previous methods like [[72]] with good performance on R@K unfairly cater to “head” predicates, e.g., on, while neglect the “tail” ones, e.g., predicates like parked on, laying on have embarrassingly 0.0 Recall@100. To speak for the valuable “tail” rather than the trivial “head”, we adopted a recent replacement, mean Recall@K (mR@K), proposed by Tang et al.[[56]] and Chen et al.[[6]]. mR@K retrieves each predicate separately and then averages R@K for all predicates.
Citation
@misc{tang2020unbiased,
title={Unbiased Scene Graph Generation from Biased Training},
author={Tang et al. (2020)},
year={2020},
note={arXiv:2002.11949}
}
- arXiv: 2002.11949