qhjqhj00 enem-vision-eval
Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams — Pires et al. (2023) (arXiv:2311.14169, 2023)
What this evaluates
Evaluates multimodal and text-only language models on Brazilian university admission exams (ENEM), specifically probing their ability to comprehend visual information, interpret tables/figures, and perform mathematical reasoning in a multiple-choice format.
Datasets
- ENEM 2022/2023 — total ?; splits: test (-1); repo https://github.com/piresramon/gpt-4-enem
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.
Input / output format
Input: Multiple-choice question text with five options (A–E), optionally accompanied by an image or a human-generated textual caption. Prompts are structured as 3-shot Chain-of-Thought examples.
Output: Step-by-step reasoning (Chain-of-Thought) followed by the final selected answer choice.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p.strip().upper() == g.strip().upper())
return (correct / len(gold)) * 100
Common pitfalls
- Failing to replicate the exact 3-shot Chain-of-Thought prompt template, which is critical for model performance.
- Confusing the three experimental conditions (no image, direct image, caption) that isolate different vision-language capabilities.
- Assuming human-generated captions fully preserve the visual information required for complex mathematical or table-based questions.
Evidence (verbatim from paper)
We performed three experiments to analyze multiple-choice questions: without images, with images, and with captions. Each experiment is described hereafter. 1) Without Images: In this experiment, we analyzed multiple-choice questions excluding any visual elements. ... 2) With Images: For this experiment, we aim to evaluate multimodal LMs by embedding the images directly into the question body. ... 3) With Captions: In this experiment, our focus was on assessing textual LMs by replacing images with textual descriptions containing all essential information necessary for understanding the question.
Citation
@misc{pires2023enem,
title={Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams},
author={Pires et al. (2023)},
year={2023},
note={arXiv:2311.14169}
}
- arXiv: 2311.14169