Scigenbench Eval

scigenbench-eval

Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility — Lin et al. (2026) (arXiv:2601.17027, 2026)

What this evaluates

Evaluates the logical correctness, structural fidelity, and information utility of AI-generated scientific images. It probes whether generated visuals accurately encode domain-specific facts and geometric relationships, and whether they are indispensable for solving visually grounded scientific quizzes.

Datasets

• SciGenBench — total ?; splits: test (-1)

Metrics

• inverse_validation_rate (primary) — range: percent
  • Proportion of images in the evaluation set for which a VQA model correctly answers all associated atomic quizzes. Formula: (1/|D|) * sum_{I in D} I(sum_{q in Q_I} V(I,q) == |Q_I|).
• lmm_as_judge_score — range: other
  • Score from 0 to 2 per dimension across five criteria: Correctness & Fidelity, Layout & Precision, Readability & Occlusion, Scientific Plausibility, and Expressiveness & Richness. Evaluated by Gemini-3-Flash with a reasoning critique.

Input / output format

Input: Text instruction describing a scientific concept or structure; for reference-based metrics, ground-truth images are also provided.

Output: Generated scientific image. For evaluation, the model's output is scored by an LMM judge (0-2 per dimension) and tested against atomic quizzes via a VQA engine.

Scoring recipe

def compute_inverse_validation_rate(generated_images, quizzes, vqa_model):
    correct_count = 0
    for img in generated_images:
        img_quizzes = quizzes[img]
        all_correct = all(vqa_model(img, q) for q in img_quizzes)
        if all_correct:
            correct_count += 1
    return (correct_count / len(generated_images)) * 100

Common pitfalls

• Standard pixel-level metrics (FID, SSIM, PSNR) are explicitly noted as unreliable for scientific diagrams due to sparse pixel distributions and should only be treated as auxiliary references on the SeePhys subset.
• Quizzes that can be answered correctly without visual input (text leakage) must be filtered out via blind solving; otherwise, the inverse validation rate will overestimate the image's information utility.

Evidence (verbatim from paper)

We define the inverse validation rate ($\mathcal{R}{\text{inv}}$) as the proportion of images for which all associated quizzes are answered correctly: $\mathcal{R}{\text{inv}}=\frac{1}{|\mathcal{D}|}\sum_{I\in\mathcal{D}}\mathbb{I}\left(\sum_{q\in\mathcal{Q}{I}}\mathcal{V}(I,q)=|\mathcal{Q}{I}|\right)$, where $\mathbb{I}(\cdot)$ is the indicator function and $\mathcal{D}$ denotes the evaluation set.

Citation

@misc{lin2026scigenbench,
  title={Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility},
  author={Lin et al. (2026)},
  year={2026},
  note={arXiv:2601.17027}
}

• arXiv: 2601.17027