Genexam Eval

genexam-eval

GenExam: A Multidisciplinary Text-to-Image Exam — Wang et al. (2025) (arXiv:2509.14232, 2025)

What this evaluates

Evaluates a model's ability to generate images that accurately reflect complex, multidisciplinary textual prompts. It probes semantic correctness, visual plausibility (spelling, logical consistency, readability), and the integration of domain knowledge with reasoning during image generation.

Datasets

• GenExam — total 1000; splits: test (1000); repo https://github.com/OpenGVLab/GenExam

Metrics

• strict score (primary) — range: percent
  • Binary or percentage score indicating whether the generated image perfectly matches all ground-truth scoring points and passes all four evaluation dimensions (semantic correctness, spelling, logical consistency, readability).
• relaxed score — range: percent
  • A lenient scoring metric that accounts for partial correctness and overall visual plausibility, allowing differentiation between models that fail strict criteria.
• semantic correctness — range: [0, 1]
  • Score measuring how accurately the image represents the core subject matter and factual content of the prompt.
• spelling — range: [0, 2]
  • Score evaluating the accuracy of text rendered within the image.
• logical consistency — range: [0, 2]
  • Score assessing whether spatial relationships, diagrams, and logical structures in the image are coherent.
• readability — range: [0, 2]
  • Score measuring the clarity and legibility of visual elements and text in the generated image.

Input / output format

Input: Text prompt (exam-style question across 10 subjects), ground-truth reference image, and fine-grained scoring rubric.

Output: Generated image.

Scoring recipe

def evaluate(prompt, gt_image, rubric, generated_image):
    semantic = judge_semantic(prompt, gt_image, generated_image)  # 0-1
    spelling = judge_spelling(generated_image)  # 0-2
    logic = judge_logic(generated_image)  # 0-2
    readability = judge_readability(generated_image)  # 0-2
    strict = 1.0 if (semantic >= 1.0 and spelling == 2 and logic == 2 and readability == 2) else 0.0
    relaxed = weighted_average(semantic, spelling, logic, readability)
    return {'strict': strict, 'relaxed': relaxed, 'semantic': semantic, 'spelling': spelling, 'logic': logic, 'readability': readability}

Common pitfalls

• Relying solely on generic metrics like CLIP Score or VQA fails to capture multidisciplinary correctness and logical consistency.
• Evaluator model choice drastically impacts results; weaker MLLMs yield correlations with no statistical significance.
• Strict scores are extremely low (<15%), so relying only on them obscures model differences; relaxed scores or dimension-wise analysis are required.

Evidence (verbatim from paper)

The strict and relaxed scores on GenExam are provided in Tab.[2]. For strict scores, we observe that all models struggle in multidisciplinary text-to-image exams, with all strict scores lower than 15% and most models nearly 0%, despite their superior capability in general T2I tasks. This demonstrates the great challenge of our benchmark. ... We then use the overall rating to calculate correlations (Kendall’s τ, Spearman’s ρ and Pearson’s r) between human scores and four automatic metrics: our relaxed score, our semantic correctness, VQA score, and CLIP score.

Citation

@misc{wang2025genexam,
  title={GenExam: A Multidisciplinary Text-to-Image Exam},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.14232}
}

• arXiv: 2509.14232