Psgg Metrics Eval

psgg-metrics-eval

A Review and Efficient Implementation of Scene Graph Generation Metrics — Lorenz et al. (2024) (arXiv:2404.09616, 2024)

What this evaluates

Evaluates panoptic scene graph generation models on their ability to predict object triplets with correct masks and relations. It probes recall-based performance at different top-k limits, mean recall across predicates, pair recall, and predicate ranking accuracy.

Datasets

• PSG — total ?; splits: test (-1)

Metrics

• Recall@k — range: [0, 1]
  • Fraction of ground truth triplets matched within the top-k predicted triplets. k can be absolute (e.g., 20, 50, 100) or relative to the number of ground truth annotations (e.g., ×1, ×10).
• Mean Recall@k (primary) — range: [0, 1]
  • Average of Recall@k computed separately for each predicate class, then averaged across all predicates. Higher values indicate better overall triplet prediction quality.
• Pair Recall@k — range: [0, 1]
  • Fraction of ground truth subject-predicate pairs matched in the top-k predictions, ignoring the object entity. Probes relation prediction quality independently of object detection.
• Predicate Rank — range: other
  • Average rank of the correct predicate for each subject-object pair in the predicted list. Lower values indicate better predicate ranking.

Input / output format

Input: Image, ground truth panoptic scene graph annotations (objects, masks, predicates, relations), and model-predicted scene graph triplets with associated masks and predicates.

Output: Model outputs a list of predicted triplets (subject, predicate, object) with corresponding masks/bounding boxes. Evaluation converts these to a standardized format for metric computation.

Scoring recipe

def compute_recall(preds, gold, k):
    top_k = preds[:k]
    matches = sum(1 for p in top_k if p in gold)
    return matches / len(gold)

def compute_mean_recall(preds, gold, k):
    recalls = []
    for pred_type in unique_predicates(gold):
        recalls.append(compute_recall(
            [t for t in preds if t.predicate == pred_type],
            [t for t in gold if t.predicate == pred_type], k))
    return sum(recalls) / len(recalls)

# k can be absolute int or relative multiplier (e.g., len(gold) * 10)

Common pitfalls

• Confusing absolute k (fixed number like 20, 50, 100) with relative k (multiplier of ground truth count), which breaks cross-dataset comparability.
• Assuming high Pair Recall@k automatically implies good Predicate Rank; the paper shows they can be independent or inversely correlated.
• Failing to support overlapping masks in output formats, which breaks evaluation for modern panoptic scene graph models that require specific pickle formats.

Evidence (verbatim from paper)

Table 1 shows Recall@k values for different k. Apart from the common k ∈ {20,50,100}, we include two relative values for k. For R@×10, it is allowed to select 10 triplets per ground truth annotation. For R@×1, the number of output triplets has to be the same as the number ground truth triplets.

Citation

@misc{lorenz2024sgbench,
  title={A Review and Efficient Implementation of Scene Graph Generation Metrics},
  author={Lorenz et al. (2024)},
  year={2024},
  note={arXiv:2404.09616}
}

• arXiv: 2404.09616