qhjqhj00 visual-genome-sgg-eval
Adaptive Self-training Framework for Fine-grained Scene Graph Generation — Kim et al. (2024) (arXiv:2401.09786, 2024)
What this evaluates
Evaluates fine-grained scene graph generation by predicting subject-predicate-object triplets from images. It measures recall and F1 scores across head, body, and tail predicate classes to assess performance on long-tailed distributions and missing annotations.
Datasets
- Visual Genome — total ?; splits: test (-1)
Metrics
R@K— range: percent- Recall@K: percentage of ground-truth predicates among the top-K predicted predicates per image.
mR@K(primary) — range: percent- Mean Recall@K: macro-average of recall across all predicate classes at top-K predictions.
F@K(primary) — range: percent- F1@K: harmonic mean of precision and recall at top-K predictions.
Input / output format
Input: Image, bounding boxes for subjects and objects, and ground-truth predicate labels (for PredCls/SGCls) or only subject/object boxes (for SGDet).
Output: Top-K predicted predicate labels for each subject-object pair.
Scoring recipe
def compute_recall_at_k(preds, gold, k):
correct = sum(1 for p, g in zip(preds[:k], gold) if p == g)
return correct / len(gold) * 100
def compute_mR_at_k(preds_list, gold_list, k):
recalls = [compute_recall_at_k(p, g, k) for p, g in zip(preds_list, gold_list)]
return sum(recalls) / len(recalls)
def compute_F1_at_k(preds, gold, k):
tp = sum(1 for p, g in zip(preds[:k], gold) if p == g)
fp = k - tp
fn = len(gold) - tp
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * prec * rec / (prec + rec) * 100 if (prec + rec) > 0 else 0
Common pitfalls
- mR@K is a macro-average, heavily penalizing models on tail predicates.
- F@K balances precision and recall, so high recall alone does not guarantee a high F-score.
- Metrics are reported at K=50 and K=100; results are not directly comparable across different K values.
Evidence (verbatim from paper)
We compare ST-SGG with state-of-the-arts methods that alleviate the long-tailed problem on commonly used benchmark datasets, VG and OI-V6. ... Motif+ST-SGG and VCTree+ST-SGG improve their performance in terms of mR@K and F@K, implying that ST-SGG greatly increases the performance on tail predicates while retaining that of head predicates.
Citation
@misc{kim2024adaptive,
title={Adaptive Self-training Framework for Fine-grained Scene Graph Generation},
author={Kim et al. (2024)},
year={2024},
note={arXiv:2401.09786}
}
- arXiv: 2401.09786