Medq Deg Eval

medq-deg-eval

MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations — Liu et al. (2026) (arXiv:2603.07769, 2026)

What this evaluates

Evaluates multimodal large language models' robustness and metacognitive reliability when processing medical images with various quality degradations (e.g., blur, noise, motion, artifacts) across different clinical capability dimensions.

Datasets

• MedQ-Deg — total 24894; splits: test (24894)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered clinical questions. Computed as the number of correct predictions divided by the total number of QA pairs.
• Calibration Shift — range: other
  • Measures the change in confidence-accuracy alignment between clean (L0) and degraded (L1/L2) images, quantifying the AI Dunning-Kruger effect where models maintain high confidence despite accuracy drops.

Input / output format

Input: A medical image paired with a clinical question requiring a text answer.

Output: A text response answering the clinical question, optionally accompanied by a confidence score.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100

Common pitfalls

• Performance is reported as a drop from clean (L0) to degraded (L1/L2) baselines, so clean accuracy must be tracked separately to compute the delta.
• Degradations are synthetically simulated rather than captured from real clinical devices, though distribution alignment is validated.
• Confidence scores used for Calibration Shift may not be natively output by all MLLMs and might require prompting or logit extraction.

Evidence (verbatim from paper)

Average accuracy across the mid-level capability dimensions, computed over L1 and L2 degradations. Numbers in parentheses show performance drop from L0 to L1&L2.

Citation

@misc{liu2026medqdeg,
  title={MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2603.07769}
}

• arXiv: 2603.07769