Sgg Vg150 Eval

sgg-vg150-eval

Compile Scene Graphs with Reinforcement Learning — Chen et al. (2025) (arXiv:2504.13617, 2025)

What this evaluates

Evaluates a model's ability to generate structured scene graphs from images without predefined object boxes. It probes visual relationship reasoning, object detection accuracy, and the model's capacity to produce structurally valid outputs under strict spatial and categorical matching criteria.

Datasets

• VG150 — total 61224; splits: train (56224), val (5000)
• PSG — total 48749; splits: train (46563), test (2186)

Metrics

• Recall (primary) — range: percent
  • Fraction of ground-truth triplets correctly predicted. A triplet matches if subject and object bounding boxes have IoU ≥ 0.5 with ground truth, and subject category, object category, and relationship label exactly match.
• mRecall — range: percent
  • Mean of Recall scores computed independently for each relation category.
• AP@50 — range: percent
  • Average Precision for object detection at an IoU threshold of 0.5.
• Failure Rate — range: percent
  • Percentage of samples where the model fails to produce a structurally valid scene graph (e.g., invalid JSON or missing required fields).

Input / output format

Input: Image paired with a prompt template that includes predefined object classes and relation categories (or omitted in ablation settings).

Output: Structured scene graph in JSON format containing subject/object bounding boxes, categories, and relationship labels.

Scoring recipe

def compute_recall(pred, gt):
    correct = 0
    for gt_trip in gt:
        for pred_trip in pred:
            if (iou(pred.sub_box, gt_trip.sub_box) >= 0.5 and
                iou(pred.obj_box, gt_trip.obj_box) >= 0.5 and
                pred.sub_cat == gt_trip.sub_cat and
                pred.obj_cat == gt_trip.obj_cat and
                pred.rel == gt_trip.rel):
                correct += 1
                break
    return correct / len(gt) if gt else 0.0
def compute_mrecall(recalls_per_cat):
    return sum(recalls_per_cat) / len(recalls_per_cat)

Common pitfalls

• Failure Rate is highly sensitive to format consistency; zero-shot M-LLMs often output invalid JSON, inflating this metric despite correct semantic predictions.
• mRecall is averaged over relation categories rather than images, which can obscure performance drops on rare or long-tail relationships.
• The SGDET protocol explicitly requires generating graphs without predefined object boxes, unlike detection-based baselines that use external detectors.
• IoU threshold for box matching is strictly 0.5; predictions with lower spatial overlap are penalized even if semantically plausible.

Evidence (verbatim from paper)

SGDET requires the model to generate scene graphs directly from the image without any predefined object boxes. Performance is evaluated using Recall and mean Recall (mRecall). Recall is computed for each image-graph pair, where a predicted triplet is considered correct if both the subject and object bounding boxes have an Intersection over Union (IoU) of at least 0.5 with the corresponding ground-truth boxes, and the subject category, object category, and relationship label all match the ground truth. Mean Recall (mRecall) is obtained by averaging the Recall across all relation categories.

Citation

@misc{chen2025compiles,
  title={Compile Scene Graphs with Reinforcement Learning},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2504.13617}
}

• arXiv: 2504.13617