Mmxu Test Eval

mmxu-test-eval

MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression — Linjie Mu et al. (2025) (arXiv:2502.11651, 2025)

What this evaluates

Evaluates multi-modal vision-language models on their ability to perform visual question answering across two temporal X-ray images to detect regional disease progression. It probes temporal reasoning, subtle change detection, and bias mitigation in medical imaging diagnostics.

Datasets

• MMXU-test — total 3000; splits: test (3000); repo https://github.com/linjiemu/MMXU

Metrics

• accuracy (primary) — range: [0, 1]
  • Accuracy is calculated as the proportion of correctly answered single-choice questions out of the total number of questions. It is reported overall and stratified by disease progression category: Worsen, Improved, and No change.

Input / output format

Input: Two X-ray images (representing different time points) paired with a single-choice question about regional disease progression or patient history.

Output: A single categorical answer choice corresponding to the correct option for the question.

Scoring recipe

def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Models exhibit strong bias toward predicting 'Worsen' outcomes, especially smaller open-source models, which inflates overall accuracy if not stratified.
• Single-image models are evaluated by concatenating two X-rays, which may degrade performance compared to native multi-image architectures.
• Human expert evaluation shows variability (e.g., Expert 3 at 90.6%), indicating potential ambiguity in some progression questions.

Evidence (verbatim from paper)

Since all the questions from our benchmark are single-choice, we use accuracy as the metric. The data from the MMXU-test benchmark was randomly divided into five parts, containing 500, 500, 500, 750, and 750 questions, respectively. We ensured that the three question categories were distributed as evenly as possible within each subset.

Citation

@misc{mu2025mmxu,
  title={MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression},
  author={Linjie Mu et al. (2025)},
  year={2025},
  note={arXiv:2502.11651}
}

• arXiv: 2502.11651