Pneumonia Xray Zero Shot Eval

pneumonia-xray-zero-shot-eval

Evaluating ChatGPT's Performance in Classifying Pneumonia from Chest X-Ray Images — Prahallad et al. (2025) (arXiv:2510.21839, 2025)

What this evaluates

This benchmark evaluates the zero-shot diagnostic capability of vision-language models on chest X-ray images for binary pneumonia detection. It probes whether models can accurately classify radiological findings without task-specific fine-tuning, relying instead on prompt engineering and pre-trained visual reasoning.

Datasets

• Chest radiographic Images (Pneumonia) — total 5863; splits: train (5216), test (624), val (16)

Metrics

• accuracy (primary) — range: [0, 1]
  • Calculated as the number of correctly classified images divided by the total number of test images. For binary classification, it represents the proportion of predictions matching the ground truth labels (Normal or Pneumonia).

Input / output format

Input: Raw chest X-ray image files from the test set.

Output: Binary classification label: either 'Normal' or 'Pneumonia'.

Scoring recipe

def calculate_accuracy(predictions, ground_truth):
    correct = sum(1 for pred, true in zip(predictions, ground_truth) if pred == true)
    return correct / len(ground_truth)

Common pitfalls

• The dataset exclusively contains pediatric patients (ages 1-5), so results do not generalize to adult chest X-rays.
• Zero-shot performance is highly sensitive to prompt design; varying prompt styles (e.g., reasoning-based vs. concise) drastically changes accuracy.
• The test set is imbalanced (390 Pneumonia vs. 234 Normal), which can skew accuracy if not accounted for.

Evidence (verbatim from paper)

This study evaluates OpenAI’s gpt-4o model in zero-shot classification of chest X-rays for pneumonia using four prompt designs, revealing that concise, feature-focused prompts achieve 74% accuracy—outperforming reasoning-based prompts—highlighting the critical role of prompt engineering in visual-language model performance for medical diagnosis.

Citation

@misc{prahallad2025evaluating,
  title={Evaluating ChatGPT's Performance in Classifying Pneumonia from Chest X-Ray Images},
  author={Prahallad et al. (2025)},
  year={2025},
  note={arXiv:2510.21839}
}

• arXiv: 2510.21839