Fine T2i Eval

fine-t2i-eval

Fine-T2I: An Open, Large-Scale, and Diverse Dataset for High-Quality T2I Fine-Tuning — Xu Ma et al. (2026) (arXiv:2602.09439, 2026)

What this evaluates

Probes the effectiveness of a large-scale text-to-image fine-tuning dataset in improving generation quality, text-image alignment, and instruction following across different model architectures (diffusion and autoregressive).

Datasets

• Artificial Analysis Image Arena (Eval Subset) — total 500; splits: test (500)

Metrics

• human_win_rate (primary) — range: [0, 1]
  • Percentage of times the fine-tuned model's generation is preferred over the baseline model's generation in pairwise comparisons by human annotators, reported separately for visual quality and text-image alignment.
• GenEval — range: [0, 1]
  • Standard automatic benchmark measuring text-image alignment across categories including single object, two objects, counting, colors, positions, and attributes. Overall score is the mean across categories.

Input / output format

Input: Text prompt (randomly sampled from the Artificial Analysis Image Arena leaderboard)

Output: Generated image

Scoring recipe

# Human Preference Win Rate
wins = 0
total = 0
for prompt in eval_prompts:
    img_base = model_base.generate(prompt)
    img_ft = model_ft.generate(prompt)
    if human_prefers(img_ft, img_base):
        wins += 1
    total += 1
human_win_rate = wins / total

# GenEval Score
geneval_scores = []
for prompt in eval_prompts:
    img = model.generate(prompt)
    geneval_scores.append(geneval.evaluate(img, prompt))
overall_geneval = mean(geneval_scores)

Common pitfalls

• Over-reliance on GenEval, which the authors explicitly note is misaligned with human preference and has limited coverage for assessing fine-tuning benefits.
• Assuming the 500 evaluation prompts are drawn from the training dataset; they are randomly sampled from a public leaderboard to ensure diversity and prevent data leakage.
• Treating visual quality and text-image alignment as a single metric; the human evaluation reports separate win rates for each dimension.

Evidence (verbatim from paper)

To better reflect real-world usage and human judgment, we therefore construct an evaluation suite by randomly sampling 500 public prompts from the Artificial Analysis Image Arena leaderboard. These prompts cover diverse user requests and have been widely used to compare the leading T2I models. We conduct large-scale human preference evaluation on the resulting generations, focusing on text-image alignment and overall human-preferred visual quality. For completeness, we also report GenEval results in Table 4, which provide a reference point for improvements on established automatic protocols.

Citation

@misc{ma2026fine_t2i,
  title={Fine-T2I: An Open, Large-Scale, and Diverse Dataset for High-Quality T2I Fine-Tuning},
  author={Xu Ma et al. (2026)},
  year={2026},
  note={arXiv:2602.09439}
}

• arXiv: 2602.09439