qhjqhj00 mmoral-bench-eval
Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis — Jing Hao et al. (2025) (arXiv:2509.09254, 2025)
What this evaluates
Evaluates large vision-language models' ability to interpret panoramic dental X-rays. It probes fine-grained anatomical recognition, pathology detection, and clinical report generation across multiple question types.
Datasets
- MMOral-Bench — total 20563; splits: test (-1); repo https://github.com/isbrycee/OralGPT
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered instances out of the total number of instances in a given category (e.g., Teeth, Patho, Overall).
Input / output format
Input: Panoramic X-ray image paired with a text question or instruction.
Output: Text response. Closed-ended questions expect short/selected answers; open-ended questions expect free-form text or clinical reports.
Scoring recipe
def calculate_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
return (correct / len(gold)) * 100
Common pitfalls
- Proprietary models frequently refuse to answer due to safety/ethical filters rather than inability.
- Open-ended VQA tasks consistently yield much lower scores than closed-ended tasks across nearly all models.
- Models show strong bias toward coarse anatomical structures (e.g., Jaw) while struggling with fine-grained details (e.g., Teeth, Pathology).
Evidence (verbatim from paper)
Current universal medical LVLMs achieve less than 40% average accuracy on MMOral-Bench, with HealthGPT-XL32 attaining peak performance at 39.59%.
Citation
@misc{hao2025towardsbetterdentalai,
title={Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis},
author={Jing Hao et al. (2025)},
year={2025},
note={arXiv:2509.09254}
}
- arXiv: 2509.09254