Panoptic Scene Graph Generation Eval

panoptic-scene-graph-generation-eval

Panoptic Scene Graph Generation — Yang et al. (2022) (arXiv:2207.11247, 2022)

What this evaluates

Evaluates a model's ability to jointly perform panoptic segmentation and scene graph generation by predicting object-background masks and relational triplets from a single image. It probes comprehensive scene understanding, accurate object grounding, and context-aware relation prediction without relying on separate detection heads.

Datasets

• PSG dataset — total 49000; splits: train (-1), val (-1), test (-1); repo https://github.com/Jingkang50/OpenPSG

Metrics

• Recall (R)@K — range: percent
  • Percentage of ground-truth triplets correctly matched by the top-K predicted triplets. K is typically 20, 50, or 100.
• Mean Recall (mR)@K (primary) — range: percent
  • Mean of recall scores across all predicate classes for the top-K predictions. Used to measure unbiased relation prediction performance, as emphasized in the paper for evaluating one-stage models.
• Panoptic Quality (PQ) — range: [0, 1]
  • Standard panoptic segmentation metric combining segmentation quality and recognition quality, reported for visualization and segmentation evaluation.

Input / output format

Input: RGB image with associated panoptic segmentation masks (instance masks + stuff masks) and ground-truth relational triplets.

Output: Predicted panoptic segmentation masks and a ranked list of relational triplets (subject, predicate, object) with confidence scores.

Scoring recipe

def compute_recall(preds, gold, k=20):
    top_k = preds[:k]
    matched = sum(1 for g in gold if g in top_k)
    return (matched / len(gold)) * 100 if gold else 0

def compute_mr(preds, gold, k=20):
    recalls = []
    for pred_class in unique_predicates(gold):
        recalls.append(compute_recall(
            [t for t in preds if t.predicate == pred_class],
            [t for t in gold if t.predicate == pred_class], k))
    return sum(recalls) / len(recalls) if recalls else 0

Common pitfalls

• Predicate classification (PredCls) metrics use ground-truth segmentation, artificially inflating performance compared to detection-based (SGDet) settings.
• One-stage models lack a separate predicate classification head, so predicate metrics are inapplicable and marked as '-', only SGG metrics are reported.
• Training schedule heavily impacts one-stage models; PSGTR requires 60 epochs to reach SOTA, while two-stage models are evaluated at 12 epochs by default.

Evidence (verbatim from paper)

Table 2 reports the scene graph generation performance of all the methods mentioned in Sec. 4.1, Sec. 4.2, and Sec. 4.3 under the PSG dataset. Table 2: Comparison between all baselines and PSGFormer. Recall (R) and mean recall (mR) are reported. IMP [64] (CVPR'17), MOTIFS [71] (CVPR'18), VC-Tree [58] (CVPR'19), and GPSNet [44] (CVPR'20) all originate from the SGG task and are adapted for the PSG task. ... Models are trained using 12 epochs by default.

Citation

@misc{yang2022panoptic,
  title={Panoptic Scene Graph Generation},
  author={Yang et al. (2022)},
  year={2022},
  note={arXiv:2207.11247}
}

• arXiv: 2207.11247