Figurebench Eval

figurebench-eval

AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations — Zhu et al. (2026) (arXiv:2602.03828, 2026)

What this evaluates

Evaluates the ability of text-to-illustration models to generate publication-ready scientific figures that balance structural fidelity, visual aesthetics, and communicative clarity based on long-form scientific text.

Datasets

• FigureBench — total 3300; splits: test (-1); repo https://github.com/ResearAI/AutoFigure

Metrics

• Overall score (primary) — range: [0, 5]
  • Composite 1–5 Likert scale rating assessing visual design excellence, communication effectiveness, and content fidelity.
• Win-Rate — range: percent
  • Percentage of times a model's generated figure is preferred over a baseline in blind pairwise comparisons.

Input / output format

Input: Natural language text or instructions extracted from scientific documents (blogs, surveys, textbooks, or papers) describing a figure to be generated.

Output: Generated scientific illustration (image, SVG, or HTML code) representing the described figure.

Scoring recipe

# For Overall Score (automated or human)
scores = [rating for rating in ratings if 1 <= rating <= 5]
overall_score = sum(scores) / len(scores)

# For Win-Rate
wins = sum(1 for pred, baseline in pairwise_comparisons if pred > baseline)
win_rate = (wins / len(pairwise_comparisons)) * 100

Common pitfalls

• Win-Rate is derived from blind pairwise comparisons against baselines, not absolute scoring, so it cannot be directly compared to Likert scale scores.
• The 1–5 Likert scale measures composite design/communication/fidelity, not standard accuracy or FID/KID metrics used in general image generation.
• Human evaluation was restricted to 10 domain experts assessing only 21 of their own first-author publications, limiting generalizability to the full 3,300-item benchmark.

Evidence (verbatim from paper)

As detailed in Table 2, AUTOFIGURE achieves the highest Overall score across all four document categories: Blog (7.60), Survey (6.99), Textbook (8.00), and Paper (7.03). Notably, AUTOFIGURE also dominates in Win-Rate evaluations through blind pairwise comparisons, achieving $75.0%$ for Blog, $78.1%$ for Survey, an exceptional $97.5%$ for Textbook, and $53.0%$ for Paper.

Citation

@misc{zhu2026autofigure,
  title={AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations},
  author={Zhu et al. (2026)},
  year={2026},
  note={arXiv:2602.03828}
}

• arXiv: 2602.03828