Igenbench Eval

igenbench-eval

IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation — Tang et al. (2026) (arXiv:2601.04498, 2026)

What this evaluates

Probes the reliability of text-to-infographic generation models by decomposing visual fidelity into atomic yes/no checks. It evaluates whether generated images accurately encode data, follow structural constraints, and maintain consistency across multiple verification questions.

Datasets

• IGenBench — total 600; splits: test (600)

Metrics

• Q-ACC (primary) — range: [0, 1]
  • Average accuracy across 10 atomic yes/no verification questions per infographic, evaluated by a multimodal LLM.
• I-ACC — range: [0, 1]
  • Proportion of infographics where all verification questions are answered correctly, measuring end-to-end correctness.

Input / output format

Input: Text prompt specifying data values, chart type, layout constraints, and accompanying verification questions.

Output: Generated infographic image.

Scoring recipe

q_accs, i_accs = [], []
for instance in dataset:
    image = generate(instance.prompt)
    q_answers = [eval_mllm(image, q) for q in instance.questions]
    q_accs.append(mean(q_answers))
    i_accs.append(all(q_answers))
Q_ACC = mean(q_accs)
I_ACC = mean(i_accs)

Common pitfalls

• High Q-ACC does not imply a reliable infographic; I-ACC is significantly lower due to long-tail failures where one critical error invalidates the whole image.
• Data encoding and completeness are consistently the hardest dimensions, often failing even in top-performing models optimized for aesthetics.
• Automatic evaluation relies heavily on the specific MLLM used for verification (Gemini-2.5-Pro), which may not perfectly match human judgment in all cases.

Evidence (verbatim from paper)

As shown in Table 1, there exists a dramatic gap between Q-ACC and I-ACC across all models, with I-ACC consistently much lower. The best-performing model achieves a Q-ACC of 0.90 but only 0.49 I-ACC.

Citation

@misc{tang2026igenbench,
  title={IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation},
  author={Tang et al. (2026)},
  year={2026},
  note={arXiv:2601.04498}
}

• arXiv: 2601.04498