# Scigenbench Eval

> Evaluates the logical correctness, structural fidelity, and information utility of AI-generated scientific images. It probes whether generated visuals accurately encode domain-specific facts and geometric relationships, and whether they are indispensable for solving visually grounded scientific quizzes. Use when the user wants to benchmark on SciGenBench, or asks about evaluating this task. Reports inverse_validation_rate.

- Skill: `qhjqhj00/scigenbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/scigenbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/scigenbench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/scigenbench-eval

---


# scigenbench-eval

> Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility — Lin et al. (2026) (arXiv:2601.17027, 2026)

## What this evaluates

Evaluates the logical correctness, structural fidelity, and information utility of AI-generated scientific images. It probes whether generated visuals accurately encode domain-specific facts and geometric relationships, and whether they are indispensable for solving visually grounded scientific quizzes.

## Datasets

- **SciGenBench** — total ?; splits: test (-1)

## Metrics

- `inverse_validation_rate` **(primary)** — range: percent
  - Proportion of images in the evaluation set for which a VQA model correctly answers all associated atomic quizzes. Formula: (1/|D|) * sum_{I in D} I(sum_{q in Q_I} V(I,q) == |Q_I|).
- `lmm_as_judge_score` — range: other
  - Score from 0 to 2 per dimension across five criteria: Correctness & Fidelity, Layout & Precision, Readability & Occlusion, Scientific Plausibility, and Expressiveness & Richness. Evaluated by Gemini-3-Flash with a reasoning critique.

## Input / output format

**Input**: Text instruction describing a scientific concept or structure; for reference-based metrics, ground-truth images are also provided.

**Output**: Generated scientific image. For evaluation, the model's output is scored by an LMM judge (0-2 per dimension) and tested against atomic quizzes via a VQA engine.

## Scoring recipe

```python
def compute_inverse_validation_rate(generated_images, quizzes, vqa_model):
    correct_count = 0
    for img in generated_images:
        img_quizzes = quizzes[img]
        all_correct = all(vqa_model(img, q) for q in img_quizzes)
        if all_correct:
            correct_count += 1
    return (correct_count / len(generated_images)) * 100
```

## Common pitfalls

- Standard pixel-level metrics (FID, SSIM, PSNR) are explicitly noted as unreliable for scientific diagrams due to sparse pixel distributions and should only be treated as auxiliary references on the SeePhys subset.
- Quizzes that can be answered correctly without visual input (text leakage) must be filtered out via blind solving; otherwise, the inverse validation rate will overestimate the image's information utility.

## Evidence (verbatim from paper)

> We define the inverse validation rate ($\mathcal{R}_{\text{inv}}$) as the proportion of images for which *all* associated quizzes are answered correctly: $\mathcal{R}_{\text{inv}}\=\frac{1}{|\mathcal{D}|}\sum_{I\in\mathcal{D}}\mathbb{I}\left(\sum_{q\in\mathcal{Q}_{I}}\mathcal{V}(I,q)\=|\mathcal{Q}_{I}|\right)$, where $\mathbb{I}(\cdot)$ is the indicator function and $\mathcal{D}$ denotes the evaluation set.

## Citation

```bibtex
@misc{lin2026scigenbench,
  title={Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility},
  author={Lin et al. (2026)},
  year={2026},
  note={arXiv:2601.17027}
}
```

- arXiv: 2601.17027