Genspace Alignment Eval

genspace-alignment-eval

GenSpace: Benchmarking Spatially-Aware Image Generation — Wang et al. (2025) (arXiv:2505.24870, 2025)

What this evaluates

Evaluates how well automated metrics and VLMs align with human judgments on spatially-aware image generation tasks across nine sub-domains.

Datasets

• GenSpace Human Alignment Test Set — total 900; splits: test (900); repo https://github.com/SpatialVision/GenSpace

Metrics

• agreement (primary) — range: percent
  • Proportion of samples where the predicted category (or mapped VLM score) matches the human-annotated category ('Correct', 'Partially Correct', 'Incorrect'). Calculated as (matches / total samples) × 100.

Input / output format

Input: Generated images paired with their original text-to-image or image-editing prompts.

Output: Human annotations: 'Correct', 'Partially Correct', or 'Incorrect'. VLM baselines output continuous scores 0–100, mapped to categories for comparison.

Scoring recipe

def compute_agreement(predictions, human_labels):
    matches = sum(1 for p, h in zip(predictions, human_labels) if p == h)
    return (matches / len(human_labels)) * 100

Common pitfalls

• VLMs struggle with allocentric perspective reasoning and quantitative spatial measurement, causing low alignment scores.
• This subsection only covers the human alignment validation subset (900 samples), not the full benchmark protocol.
• Mapping continuous VLM scores to discrete human categories can obscure fine-grained spatial errors.

Evidence (verbatim from paper)

Finally, we measure how well each method aligns with human perception by comparing its accuracy against manual human labels. To align the fine-grained continuous scores with this categorical system for comparison, we map scores as follows: 0 to "Incorrect," (0, 100) to "Partially Correct," and 100 to "Correct." Results. Tab.[3] presents the comparative results of different evaluators’ alignment with human judgment. Overall, our spatial expertise pipeline and corresponding metric demonstrate a stronger correlation with human recognition. Across sub-domains, our method achieves 76.22% average agreement with manual labels, while the most advanced VLM, Gemini-2.5-Pro, achieves only 56.44%.

Citation

@misc{wang2025genspace,
  title={GenSpace: Benchmarking Spatially-Aware Image Generation},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.24870}
}

• arXiv: 2505.24870