qhjqhj00 spatial-rel-eval
Evaluating the Generation of Spatial Relations in Text and Image Generative Models — Sim Shang Hong et al. (2024) (arXiv:2411.07664, 2024)
What this evaluates
Evaluates the ability of text-to-image and large language models to accurately generate and understand spatial relationships between objects. It probes geometric scene modeling and prepositional semantics grounding by testing models on simple and complex spatial prompts using basic geometric primitives.
Datasets
- SpatialRelBench — total ?; splits: test (-1); repo https://github.com/trolommonm/SpatialRelBench
Metrics
accuracy(primary) — range: [0, 1]- Proportion of generated images rated 'A' by human annotators or GPT-4V, where 'A' indicates the spatial relationship is correct and the number/type of objects match the prompt.
Input / output format
Input: Text prompts specifying spatial relations between three basic geometric objects (sphere, cube, cylinder). Simple format: "{object1} {preposition} {object2}". Complex format: "{object1} {preposition1} {object2}, with {object3} {preposition2} {object1}".
Output: For T2I models: a generated 2D image. For LLMs: a JSON defining a 3D scene, which is then rendered in Unity to produce an image. Evaluation output: a categorical rating (A, B, C, D) with optional explanation.
Scoring recipe
def score_spatial_relation(predictions, gold):
# predictions: list of ratings (A, B, C, D) from raters
# gold: implicit (all should be 'A' if correct)
correct = sum(1 for p in predictions if p == 'A')
return correct / len(predictions)
Common pitfalls
- Ambiguity in spatial perspective (viewer-centric vs. object-centric) can cause false negatives if not standardized in the prompt.
- CLIP-based similarity metrics should be avoided as they operate as bag-of-words and conflate inverted spatial relations (e.g., 'A above B' vs 'B above A').
- LLM outputs must be rendered via Unity to match T2I evaluation conditions; direct text comparison misses spatial fidelity.
Evidence (verbatim from paper)
To ensure evaluation accuracy, control examples (with unambiguous ground-truth ratings) comprising 10% of the total examples were included. Only batches that had ≥ 87.5% (i.e. 7 out of 8) accurate ratings for the control examples were accepted.
Citation
@misc{hong2024spatialrel,
title={Evaluating the Generation of Spatial Relations in Text and Image Generative Models},
author={Sim Shang Hong et al. (2024)},
year={2024},
note={arXiv:2411.07664}
}
- arXiv: 2411.07664