Scene Bench Eval

scene-bench-eval

What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation — Chen et al. (2024) (arXiv:2411.15435, 2024)

What this evaluates

Evaluates the factual consistency and scene graph adherence of text-to-image generation models. It probes whether generated images accurately preserve specified objects and their spatial/relational configurations as defined by input scene graphs, rather than just measuring aesthetic quality or text-image alignment.

Datasets

• Visual Genome (VG) test set — total ?; splits: simple (3993), medium (930), hard (173)
• MegaSG — total 1000000; splits: test_simple (15000), test_medium (20000), test_hard (15000)

Metrics

• SGScore (primary) — range: percent
  • A scene-graph-based consistency score that measures the accuracy of object presence and relationship preservation in generated images relative to a target scene graph. Computed using a multimodal LLM to detect entities and relations in the output image and comparing them against the ground-truth scene graph triples.
• ObjectRecall — range: percent
  • Percentage of ground-truth objects from the scene graph that are correctly detected in the generated image.
• RelationRecall — range: percent
  • Percentage of ground-truth relationships (subject-predicate-object triples) from the scene graph that are correctly detected in the generated image.
• Inception Score (IS) — range: other
  • Standard generative model metric measuring image quality and diversity based on classifier confidence distributions.
• Fréchet Inception Distance (FID) — range: other
  • Standard generative model metric measuring the distance between feature distributions of real and generated images.
• CLIPScore — range: other
  • Measures alignment between image and text prompt using CLIP embeddings.

Input / output format

Input: Scene graph encoded as text triples in the format '{subject} {predicate} {object}' (e.g., 'cat sitting on desk'), or a composed natural language description derived from the scene graph.

Output: A single RGB image generated by the diffusion model.

Scoring recipe

def compute_sg_score(image, scene_graph):
    # 1. Use MLLM (Gemini 1.5 Flash) to detect objects and relations in image
    det_objs = mllm_detect_objects(image)
    det_rels = mllm_detect_relations(image)
    # 2. Parse ground-truth scene graph
    gt_objs = set(scene_graph.objects)
    gt_rels = set(scene_graph.triples)
    # 3. Calculate recall
    obj_recall = len(det_objs & gt_objs) / max(len(gt_objs), 1)
    rel_recall = len(det_rels & gt_rels) / max(len(gt_rels), 1)
    # 4. SGScore aggregates recall metrics
    sg_score = (obj_recall + rel_recall) / 2 * 100
    return sg_score

Common pitfalls

• Standard metrics like FID and CLIPScore can be misleading for factual consistency; e.g., SD v1.5 outperforms SD v2.1 on FID but scores lower on SGScore due to missing objects/relationships.
• The Visual Genome test set is heavily biased toward simple scenes, making complexity-based evaluation unreliable without a dedicated complex-scene benchmark like MegaSG.
• Image quality (FID) remains stable across complexity levels, but factual consistency (SGScore) degrades significantly, requiring separate evaluation of generation quality vs. scene adherence.

Evidence (verbatim from paper)

We employ common metrics such as Inception Score (IS), Fréchet Inception Distance (FID), and CLIPScore. Additionally, we introduce ObjectRecall, RelationRecall, and SGScore... SGScore provides much more distinguishability than other metrics like FID and CLIPScore. For instance, SD v1.5 has a better FID score than SD v2.1 (42.8 vs. 46.6), yet its SGScore is lower than that of SD v2.1 (52.5 vs. 54.4), indicating there are more missed objects and relationships in the images generated by SD v1.5.

Citation

@misc{chen2024scenebench,
  title={What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2411.15435}
}

• arXiv: 2411.15435