Genfig1 Eval

genfig1-eval

GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models — Guan et al. (2026) (arXiv:2604.04172, 2026)

What this evaluates

This benchmark evaluates vision-language models' ability to synthesize scientifically faithful, visually coherent 'Figure 1' summaries from academic paper text. It probes deep cross-modal reasoning, concept selection, spatial layout planning, and adherence to scientific content without distortion.

Datasets

• GenFig1 — total 120; splits: test (120)

Metrics

• VLM-as-a-Judge (primary) — range: [0, 100]
  • GPT-4.1 scores a generated figure against its corresponding caption on six dimensions (Clarity, Faithfulness, Info. Density, Interestingness, Legibility, Aesthetic) from 0 to 10, then rescales the average to [0,100].
• Text-Rich Catastrophic Neglect Score — range: [0, 100]
  • Extract core ideas from the figure caption via a VLM. Score each idea's presence in the generated figure as 0 (not covered), 0.5 (partially covered), or 1 (fully covered). Compute the average across all ideas and rescale to [0,100].
• DINOv2 Score — range: [0, 100]
  • Compute cosine similarity between DINOv2 embeddings of the generated figure and the reference (golden) figure, then rescale the similarity score to [0,100].

Input / output format

Input: Textual description of an academic paper, including title, abstract, introduction, and figure caption.

Output: A generated Figure 1 image or SVG code representing a visual summary of the provided text.

Scoring recipe

def compute_neglect_score(predictions, gold):
    core_ideas = extract_core_ideas_from_caption(gold['caption'])
    idea_scores = []
    for idea in core_ideas:
        score = vlm_assess_idea_in_figure(idea, predictions['figure'])
        idea_scores.append(score)
    raw_avg = sum(idea_scores) / len(idea_scores)
    return rescale_to_100(raw_avg)

Common pitfalls

• Step-wise decomposition prompting (CoT/CoI) consistently underperforms zero-shot baselines, contrary to typical LLM/VLM expectations.
• Text-to-SVG generation pipelines significantly lag behind text-to-image models due to difficulties in spatial constraint satisfaction and canvas size limitations.
• Most automated metrics (except VLM-as-a-Judge) show only mild to fair correlation with human preference rankings.

Evidence (verbatim from paper)

Following the DinoScore proposed in StarVector Rodriguez et al. (2023a), we adopt the DINOv2 Score, which leverages DINOv2 embeddings Oquab et al. (2023) and cosine similarity to assess visual similarity between the generated figure and the golden (reference) figure.

Citation

@misc{guan2026genfig1,
  title={GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models},
  author={Guan et al. (2026)},
  year={2026},
  note={arXiv:2604.04172}
}

• arXiv: 2604.04172