Medq Bench Eval

medq-bench-eval

MedQ-Bench: Evaluating and Exploring Medical Image Quality Assessment Abilities in MLLMs — Liu et al. (2025) (arXiv:2510.01691, 2025)

What this evaluates

Probes multimodal large language models' ability to assess medical image quality through low-level visual attribute detection and no-reference or comparative reasoning. It evaluates how well models identify image degradations, describe clinical attributes, and compare quality across different imaging modalities.

Datasets

• MedQ-Bench — total 3308; splits: dev (-1), test (-1); repo https://github.com/liujiyaoFDU/MedQBench

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions across Yes-or-No, What, and How subtasks in the perception phase.

Input / output format

Input: A medical image paired with a task-specific prompt (multiple-choice question for perception; open-ended or structured prompt for no-reference or comparative quality reasoning).

Output: For perception: a single selected option from the provided choices. For reasoning: a generated text response describing image quality attributes, which is subsequently scored by an automated rubric.

Scoring recipe

perception_correct = sum(1 for pred, gold in perception_preds if pred == gold)
perception_acc = (perception_correct / len(perception_preds)) * 100

# Reasoning scoring (automated rubric validated against humans)
comp = score_dimension(predictions, gold, 'completeness') # 0-2
prec = score_dimension(predictions, gold, 'preciseness') # 0-2
cons = score_dimension(predictions, gold, 'consistency') # 0-2
qual = score_dimension(predictions, gold, 'quality')     # 0-2
reasoning_overall = comp + prec + cons + qual

Common pitfalls

• Assuming medical-specialized MLLMs inherently outperform generalist models on IQA tasks (the study shows the opposite).
• Overestimating model robustness to subtle artifacts; performance actually drops for mild degradations compared to severe ones.
• Treating reasoning outputs as binary correct/incorrect; they require multi-dimensional rubric scoring (0–2 per dimension) rather than exact-match accuracy.

Evidence (verbatim from paper)

The confusion matrices in the appendix (Figure[13]) demonstrate strong alignment between human expert scores and GPT-4o automated evaluation across all three dimensions, with consistently high accuracy rates: 83.3% for completeness, 87.0% for preciseness, and 90.5% for consistency, with all individual class recall rates exceeding 80%.

Citation

@misc{liu2025medqbench,
  title={MedQ-Bench: Evaluating and Exploring Medical Image Quality Assessment Abilities in MLLMs},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2510.01691}
}

• arXiv: 2510.01691