T2i Corebench Eval

t2i-corebench-eval

Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play? — Li et al. (2025) (arXiv:2509.03516, 2025)

What this evaluates

Evaluates text-to-image models' ability to handle high compositional density and multi-step visual reasoning. It probes instance, attribute, and relation binding, text rendering, and deductive/inductive/abductive inference capabilities.

Datasets

• T2I-CoReBench — total ?; splits: test (-1)

Metrics

• Overall Score (primary) — range: percent
  • Mean score across all samples for both composition and reasoning dimensions. Scores are computed on a 0-100 scale by an MLLM evaluator (Gemini 2.5 Flash) judging image-prompt alignment per dimension.
• Mean Composition Score — range: percent
  • Average score across instance, attribute, relation, and text rendering dimensions.
• Mean Reasoning Score — range: percent
  • Average score across logical, background, hidden, prompt, general, attribute, constraint, and relation reasoning dimensions.

Input / output format

Input: Text prompts containing complex scene descriptions requiring multi-step inference and high compositional density.

Output: Generated images corresponding to the input prompts.

Scoring recipe

def evaluate(prompt, generated_image):
    evaluator = Gemini2.5Flash()
    dim_scores = {}
    for dim in COMPOSITION_DIMS + REASONING_DIMS:
        dim_scores[dim] = evaluator.judge_alignment(prompt, generated_image, dim)
    comp_mean = mean(dim_scores[d] for d in COMPOSITION_DIMS)
    reason_mean = mean(dim_scores[d] for d in REASONING_DIMS)
    overall = mean(comp_mean, reason_mean)
    return overall

Common pitfalls

• Prompt rewriting explicitly reasons over text before generation, which boosts reasoning scores but often drops composition scores by omitting explicit visual elements.
• Textual reasoning alone cannot fix inherent visual biases (e.g., generating circular wheels for 'square wheels' prompts), requiring multimodal interaction mechanisms beyond prompt rewriting.

Evidence (verbatim from paper)

As shown in Table 3, we evaluate a wide range of T2I models on our T2I-CoReBench, revealing valuable insights into their strengths, weaknesses, and advancements, particularly in handling real-world scenarios that require high compositional density and reasoning intensity... Despite achieving the highest overall score, Imagen 4 Ultra achieves only 72.9 in reasoning (9.5 below its composition score)... These results indicate that current T2I models still struggle to infer implicit visual elements from prompts, underscoring reasoning as the central unsolved challenge in our benchmark.

Citation

@misc{li2025easier,
  title={Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2509.03516}
}

• arXiv: 2509.03516