Unbiased Sgg Eval

unbiased-sgg-eval

Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation — Dong et al. (2022) (arXiv:2203.09811, 2022)

What this evaluates

Evaluates a model's ability to generate unbiased scene graphs by predicting pairwise relationships between objects in images. It specifically probes robustness to long-tailed predicate distributions by measuring per-class recall averaged across all predicate classes, rather than relying on global recall which favors head classes.

Datasets

• VG150 — total ?; splits: train (-1), val (5000), test (-1)
• GQA200 — total ?; splits: train (-1), val (5000), test (-1)

Metrics

• mR@K (primary) — range: percent
  • Computes Recall@K for each predicate class individually, then averages these recalls across all predicate classes. K is typically 50 or 100. This averaging mitigates bias from head classes in long-tailed datasets.

Input / output format

Input: Image, ground-truth bounding boxes and object classes (for PredCls/SGCls) or raw image (for SGDet), plus ground-truth scene graph annotations (objects, boxes, predicates).

Output: Predicted bounding boxes, object classes, and pairwise predicate relationships for each image.

Scoring recipe

def compute_mR_at_K(predictions, gold, K=50):
    per_class_recalls = []
    for pred_class in gold.predicate_classes:
        gt_indices = [i for i, g in enumerate(gold) if g.predicate == pred_class]
        pred_indices = [i for i, p in enumerate(predictions) if p.predicate == pred_class]
        correct = sum(1 for i in gt_indices if i in pred_indices[:K])
        recall = correct / len(gt_indices) if gt_indices else 0
        per_class_recalls.append(recall)
    return sum(per_class_recalls) / len(per_class_recalls) * 100

Common pitfalls

• Confusing mR@K with standard Recall@K; mR@K averages per-class recall to prevent head-class dominance in long-tailed distributions.
• Using the full VG or GQA splits instead of the specified VG150/GQA200 subsets (50/100 predicate classes).
• Mixing up PredCls, SGCls, and SGDet tasks, which differ in whether ground-truth boxes/classes are provided during inference.

Evidence (verbatim from paper)

Following [[42]], [[20]], [[17]], [[30]], [[43]], [[4]], [[29]], we use mean Recall@K (mR@K)[[31]], [[3]], which computes the average Recall@K (R@K) for each predicate class, to evaluate the unbiased SGG. As R@K is easily dominated by the head classes due to the extremely unbiased dataset, mR@K could give a fair performance appraisal for both head and tail classes, which is widely used as an unbiased evaluation metric.

Citation

@misc{dong2022stacked,
  title={Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation},
  author={Dong et al. (2022)},
  year={2022},
  note={arXiv:2203.09811}
}

• arXiv: 2203.09811