T2i Reasoning Eval

t2i-reasoning-eval

Enhanced Text-to-Image Generation by Fine-grained Multimodal Reasoning — Kim et al. (2026) (arXiv:2604.13491, 2026)

What this evaluates

Evaluates text-to-image generation models on their ability to align with complex, compositional prompts through iterative fine-grained reasoning and self-refinement. It probes capabilities in object counting, attribute binding, spatial relationships, and handling long, dense prompts.

Datasets

• GenEval — total ?; splits: test (-1)
• T2I-CompBench — total ?; splits: test (-1)
• DPGBench — total ?; splits: test (-1)

Metrics

• GenEval, T2I-CompBench, and DPGBench alignment scores (primary) — range: [0, 1]
  • Models generate images from prompts and optionally refine them up to three times based on self-judgment. Alignment is measured using official benchmark implementations (with a corrected DPGBench script for multi-GPU stability). Scores are reported as accuracy/alignment percentages per category and overall, ranging from 0 to 1.

Input / output format

Input: Text prompts ranging from simple object descriptions to long, dense compositional instructions.

Output: Generated images (iteratively refined up to 3 steps based on self-judgment).

Scoring recipe

scores = []
for prompt, gold in dataset:
    img = model.generate(prompt)
    for step in range(3):
        feedback = model.self_judge(img, prompt)
        if feedback == 'aligned': break
        img = model.refine(img, feedback)
    scores.append(benchmark_score(img, prompt))
return mean(scores)

Common pitfalls

• DPGBench official evaluation code has known bugs under multi-GPU configurations; a corrected version must be used for reliable results.
• CLIP-T score used in T2I-CompBench may not capture fine-grained semantic improvements, potentially showing marginal declines in Non-Spatial categories despite actual visual refinements.
• Iterative refinement is conditional: if the self-judge deems an image aligned, subsequent steps are skipped, which affects average scores across the dataset.

Evidence (verbatim from paper)

To assess the image reasoning capabilities of our models, we utilize three comprehensive benchmarks: DPGBench, GenEval, and T2I-CompBench. For GenEval and T2I-Compbench, we strictly follow the official implementations to ensure consistency and comparability. For these two models, we report performance across three sequential stages: the initial T2I generation (denoted as 1st) and two subsequent refinement rounds (2nd and 3rd). Specifically, if the self-judge determines an image is aligned, subsequent refinement steps are skipped and the preceding image is retained.

Citation

@misc{kim2026enhanced,
  title={Enhanced Text-to-Image Generation by Fine-grained Multimodal Reasoning},
  author={Kim et al. (2026)},
  year={2026},
  note={arXiv:2604.13491}
}

• arXiv: 2604.13491