Ophthalmic Multimodal Eval

ophthalmic-multimodal-eval

A Novel Ophthalmic Benchmark for Evaluating Multimodal Large Language Models with Fundus Photographs and OCT Images — Xiaoyi Liang et al. (2025) (arXiv:2503.07094, 2025)

What this evaluates

Evaluates multimodal large language models on clinical ophthalmic image interpretation, specifically diagnosing retinal and macular diseases from fundus photographs and optical coherence tomography (OCT) scans. It probes the models' ability to recognize normal conditions and identify specific pathological states across diverse disease categories.

Datasets

• Ophthalmic Multimodal Benchmark — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly diagnosed conditions out of total instances. The paper also reports a normalized score calculated as (model_accuracy / max_model_accuracy) * 90 for comparative visualization.

Input / output format

Input: A single fundus photograph or an optical coherence tomography (OCT) image.

Output: A textual diagnosis identifying the specific ophthalmic disease condition (e.g., diabetic retinopathy, choroidal neovascularization, normal).

Scoring recipe

def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def normalize_scores(accuracy_scores):
    max_acc = max(accuracy_scores.values())
    return {model: (acc / max_acc) * 90 for model, acc in accuracy_scores.items()}

Common pitfalls

• Accuracy drops to near zero for rare or complex conditions like choroidal neovascularization (CNV) and myopia (MYA), making aggregate metrics potentially misleading for clinical reliability.
• The paper uses a normalized scoring system (max=90) for visualization, which can obscure absolute performance differences and make cross-study comparison difficult.
• Single-label diagnosis is assumed; multi-label or multi-class confusion is not explicitly addressed in the protocol, which may inflate accuracy for models that output multiple conditions.

Evidence (verbatim from paper)

The accuracy of each model in identifying fundus images for normal condition and each retinal disease is shown in Figure 3. None of the seven multimodal large models, including GPT-4o, answered correctly in the four diseases of HTR, LS, ODP and ODC.

Citation

@misc{liang2025ophthalmicbenchmark,
  title={A Novel Ophthalmic Benchmark for Evaluating Multimodal Large Language Models with Fundus Photographs and OCT Images},
  author={Xiaoyi Liang et al. (2025)},
  year={2025},
  note={arXiv:2503.07094}
}

• arXiv: 2503.07094