Pvsg Eval

pvsg-eval

Panoptic Video Scene Graph Generation — Yang et al. (2023) (arXiv:2311.17058, 2023)

What this evaluates

Evaluates a model's ability to generate temporal scene graphs where nodes are grounded with pixel-level panoptic segmentation masks instead of bounding boxes, capturing non-rigid objects, backgrounds, and fine-grained interactions in dynamic videos.

Datasets

• PVSG — total 400; splits: train (338), test (62)

Metrics

• R/mR@20 (primary) — range: [0, 100] percent
  • Recall (R) and mean Recall (mR) at top-20 predicted relations per query. A prediction is counted as correct if the predicted mask tube overlaps with the ground-truth mask tube above a specified IoU threshold (e.g., 0.1 or 0.5).

Input / output format

Input: Video frames with ground-truth panoptic segmentation masks and temporal scene graph annotations (subject-predicate-object triplets with mask tubes).

Output: Predicted scene graph triplets with corresponding mask tubes for each frame in the video.

Scoring recipe

def compute_recall(preds, golds, k=20, iou_thresh=0.1):
    correct = 0
    for gt in golds:
        top_k = sorted(preds[gt.subject], key=lambda x: x.score, reverse=True)[:k]
        for pred in top_k:
            if compute_mask_iou(pred.mask_tube, gt.mask_tube) >= iou_thresh:
                correct += 1
                break
    return (correct / len(golds)) * 100

Common pitfalls

• The task uses mask tubes instead of bounding boxes, requiring temporal consistency across frames rather than single-frame detection.
• Recall is evaluated at a strict IoU threshold (e.g., 0.5) but the paper notes a looser 0.1 threshold yields significantly higher scores, which can mislead comparisons if not specified.
• Performance heavily depends on the first-stage mask tracking quality, as poor tracking directly degrades relation prediction regardless of the second-stage model.

Evidence (verbatim from paper)

We split the dataset with 338 videos for training and 62 videos for testing. When examining Table 2, it is crucial to prioritize the R/mR@20 as it represents our most significant indicator. The highest value for R@20 currently stands at 3.88, meaning that roughly for every 25 ground-truth triplets, one meets the criteria for a successful recall, indicating a relatively low efficiency. However, when setting the threshold to 0.1, the score improves to around 10, meaning the model can predict one in every 10 triplets with a looser requirement of recall.

Citation

@misc{yang2023panoptic,
  title={Panoptic Video Scene Graph Generation},
  author={Yang et al. (2023)},
  year={2023},
  note={arXiv:2311.17058}
}

• arXiv: 2311.17058