Egtr Sgg Eval

egtr-sgg-eval

EGTR: Extracting Graph from Transformer for Scene Graph Generation — Im et al. (2024) (arXiv:2404.02072, 2024)

What this evaluates

Evaluates a model's ability to detect objects and predict relational triplets (subject-predicate-object) in natural images. It probes both object detection accuracy and scene graph generation quality under graph constraints and standard recall/mAP metrics.

Datasets

• Visual Genome — total 88000; splits: train (57000), val (5000), test (26000)
• Open Image V6 — total 133000; splits: train (126000), val (2000), test (5000)

Metrics

• Recall@k (R@k) (primary) — range: percent
  • Class-agnostic recall of predicted predicates at top-k per object pair under a graph constraint (single predicate allowed per object pair).
• mean Recall@k (mR@k) — range: percent
  • Mean of R@k aggregated across each predicate category, emphasizing tail predicates.
• AP50 — range: percent
  • Average Precision at IoU threshold 0.5 for object detection performance across all objects in the scene.
• micro-R@50 (primary) — range: percent
  • Micro-averaged recall at top-50 predictions for Open Image V6.
• wmAP_rel — range: percent
  • Weighted mean AP for predicting subject and object bounding boxes separately.
• wmAP_phr — range: percent
  • Weighted mean AP for predicting a union bounding box of subject and object pairs.
• Final Score — range: other
  • Weighted combination: 0.2 × micro-R@50 + 0.4 × wmAP_rel + 0.4 × wmAP_phr.

Input / output format

Input: RGB images (standard resolution; FPS measured on images resized to min 600px shortest side and max 1000px longest side).

Output: Set of predicted scene graph triplets (subject bounding box, predicate class, object bounding box) and object bounding boxes.

Scoring recipe

def score_vg(preds, gold, k):
    recalls = []
    for pair in gold:
        top_k = sorted(preds[pair], key=lambda x: x.score, reverse=True)[:k]
        recalls.append(1.0 if any(p.rel == gold[pair].rel for p in top_k) else 0.0)
    R_k = sum(recalls) / len(recalls)
    mR_k = mean(mean_recall_per_predicate_class(recalls, gold))
    return R_k, mR_k

def score_oiv6(preds, gold):
    micro_R50 = recall_at_k(preds, gold, k=50, average='micro')
    wmAP_rel = weighted_mean_AP(preds.sub_boxes, preds.rel, gold)
    wmAP_phr = weighted_mean_AP(preds.union_boxes, preds.rel, gold)
    final = 0.2 * micro_R50 + 0.4 * wmAP_rel + 0.4 * wmAP_phr
    return micro_R50, wmAP_rel, wmAP_phr, final

Common pitfalls

• Graph constraint: Evaluation enforces a single predicate per object pair, which differs from standard multi-label triplet detection and can penalize models that predict multiple valid relations.
• AP50 calculation varies by architecture: For explicit object detectors, AP50 is computed directly on object boxes. For triplet detectors without explicit detectors, AP50 requires applying NMS to the union of predicted subjects and objects first.
• Logit adjustment trade-off: Applying logit adjustment significantly improves mR@k (tail predicates) but drops R@k (head predicates), requiring careful threshold tuning to balance performance.

Evidence (verbatim from paper)

We adopt Scene Graph Detection (SGDet) evaluation settings and report Recall@k (R@k) which is class agnostic and mean Recall@k (mR@k) that aggregates the recalls for each predicate category. Following Motifs [45], these metrics are measures with graph constraint, which means each object pair can have a single predicate category. ... For recall evaluation, micro-R@50 is adopted. The WMAP is evaluated with two settings: wmAP_rel for predicting boxes of subject entity and object entity separately and wmAP_phr for predicting a union box of them. The final score is calculated by 0.2 × micro-R@50 + 0.4 × wmAP_rel + 0.4 × wmAP_phr.

Citation

@misc{im2024egtr,
  title={EGTR: Extracting Graph from Transformer for Scene Graph Generation},
  author={Im et al. (2024)},
  year={2024},
  note={arXiv:2404.02072}
}

• arXiv: 2404.02072