# Unbiased Sgg Eval

> Evaluates a model's ability to generate unbiased scene graphs by predicting pairwise relationships between objects in images. It specifically probes robustness to long-tailed predicate distributions by measuring per-class recall averaged across all predicate classes, rather than relying on global recall which favors head classes. Use when the user wants to benchmark on VG150, GQA200, or asks about evaluating this task. Reports mR@K.

- Skill: `qhjqhj00/unbiased-sgg-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/unbiased-sgg-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/unbiased-sgg-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/unbiased-sgg-eval

---


# unbiased-sgg-eval

> Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation — Dong et al. (2022) (arXiv:2203.09811, 2022)

## What this evaluates

Evaluates a model's ability to generate unbiased scene graphs by predicting pairwise relationships between objects in images. It specifically probes robustness to long-tailed predicate distributions by measuring per-class recall averaged across all predicate classes, rather than relying on global recall which favors head classes.

## Datasets

- **VG150** — total ?; splits: train (-1), val (5000), test (-1)
- **GQA200** — total ?; splits: train (-1), val (5000), test (-1)

## Metrics

- `mR@K` **(primary)** — range: percent
  - Computes Recall@K for each predicate class individually, then averages these recalls across all predicate classes. K is typically 50 or 100. This averaging mitigates bias from head classes in long-tailed datasets.

## Input / output format

**Input**: Image, ground-truth bounding boxes and object classes (for PredCls/SGCls) or raw image (for SGDet), plus ground-truth scene graph annotations (objects, boxes, predicates).

**Output**: Predicted bounding boxes, object classes, and pairwise predicate relationships for each image.

## Scoring recipe

```python
def compute_mR_at_K(predictions, gold, K=50):
    per_class_recalls = []
    for pred_class in gold.predicate_classes:
        gt_indices = [i for i, g in enumerate(gold) if g.predicate == pred_class]
        pred_indices = [i for i, p in enumerate(predictions) if p.predicate == pred_class]
        correct = sum(1 for i in gt_indices if i in pred_indices[:K])
        recall = correct / len(gt_indices) if gt_indices else 0
        per_class_recalls.append(recall)
    return sum(per_class_recalls) / len(per_class_recalls) * 100
```

## Common pitfalls

- Confusing mR@K with standard Recall@K; mR@K averages per-class recall to prevent head-class dominance in long-tailed distributions.
- Using the full VG or GQA splits instead of the specified VG150/GQA200 subsets (50/100 predicate classes).
- Mixing up PredCls, SGCls, and SGDet tasks, which differ in whether ground-truth boxes/classes are provided during inference.

## Evidence (verbatim from paper)

> Following [[42]], [[20]], [[17]], [[30]], [[43]], [[4]], [[29]], we use mean Recall@K (mR@K)[[31]], [[3]], which computes the average Recall@K (R@K) for each predicate class, to evaluate the unbiased SGG. As R@K is easily dominated by the head classes due to the extremely unbiased dataset, mR@K could give a fair performance appraisal for both head and tail classes, which is widely used as an unbiased evaluation metric.

## Citation

```bibtex
@misc{dong2022stacked,
  title={Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation},
  author={Dong et al. (2022)},
  year={2022},
  note={arXiv:2203.09811}
}
```

- arXiv: 2203.09811