qhjqhj00 brazilian-medical-exam-eval
Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam — Truyts et al. (2025) (arXiv:2507.19885, 2025)
What this evaluates
Evaluates zero-shot medical knowledge and clinical reasoning of LLMs and MLLMs on a Brazilian Portuguese medical residency exam. Probes text-only comprehension versus multimodal image interpretation across five clinical domains.
Datasets
- HCFMUSP Brazilian Portuguese Medical Residency Exam — total 117; splits: test (117)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered multiple-choice questions out of the total 117 items.
processing_time— range: seconds- Mean time in seconds to generate a response per question.
safety_harm_rating— range: binary- Binary classification (safe/unsafe) of model explanations based on potential patient harm, assessed by three clinical experts using Gwet's AC1 for inter-rater agreement.
Input / output format
Input: Multiple-choice medical exam questions in Brazilian Portuguese, provided either as text-only or with accompanying medical images (radiological or non-radiological).
Output: Model selects the correct answer option from the provided choices.
Scoring recipe
def compute_metrics(predictions, gold_answers, processing_times):
correct = sum(1 for p, g in zip(predictions, gold_answers) if p == g)
accuracy = (correct / len(gold_answers)) * 100
mean_time = sum(processing_times) / len(processing_times)
return {'accuracy': accuracy, 'processing_time': mean_time}
Common pitfalls
- Models are evaluated in a strict zero-shot setting with no prompt tuning or few-shot examples, which may penalize models not optimized for direct exam-style prompting.
- Multimodal performance drops significantly for radiological images, but the evaluation does not separate vision-language alignment errors from pure medical knowledge gaps.
- Expert safety/harm assessment uses Gwet's AC1 for inter-rater reliability, but agreement is notably lower for incorrectly answered questions, introducing subjectivity in safety labeling.
Evidence (verbatim from paper)
The accuracy and processing time for Claude-3-Sonnet, Claude-3-Opus, Claude-3.5-Sonnet and Claude-3-Haiku were analyzed on all 117 questions (Table[2]). Overall, we have observed a tendency to decrease accuracy (except for Claude 3.5 sonnet) mean processing time per question increases were observed with the addiction of questions containing images (Fig.[3], filled circles).
Citation
@misc{truyts2025zeroshot,
title={Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam},
author={Truyts et al. (2025)},
year={2025},
note={arXiv:2507.19885}
}
- arXiv: 2507.19885