Mimic Cxr Medical Eval

mimic-cxr-medical-eval

Learning from All: Concept Alignment for Autonomous Distillation from Multiple Drifting MLLMs — Xiaoyu Yang et al. (2025) (arXiv:2510.04142, 2025)

What this evaluates

Evaluates the robustness, consistency, and generalization of a distilled multimodal large language model on medical imaging tasks. It probes the model's ability to perform multi-label chest disease classification and generate diagnostic radiology reports from chest X-ray images, even when trained on limited data from drifting teacher models.

Datasets

• MIMIC-CXR — total 371920; splits: train (-1), test (-1)

Metrics

• Top-1 accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted disease labels out of the total number of test instances. Evaluated as binary classification accuracy for each of the 14 diseases, then averaged across all labels.
• BLEU-4 — range: [0, 1]
  • N-gram precision metric evaluating the overlap between generated diagnostic reports and reference radiology reports, considering 4-gram matches.
• ROUGE-L — range: [0, 1]
  • Recall-oriented metric measuring the longest common subsequence overlap between generated and reference reports to assess narrative completeness.
• METEOR — range: [0, 1]
  • Metric that aligns generated and reference text using synonymy and stemming to capture lexical and semantic alignment.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between positive and negative disease labels in zero-shot settings.

Input / output format

Input: Chest X-ray image (and optional instruction prompt for report generation).

Output: For classification: predicted disease label(s) or probability scores over 14 categories. For report generation: free-text diagnostic radiology report.

Scoring recipe

def compute_avg_top1_accuracy(predictions, gold_labels, num_classes=14):
    per_class_acc = []
    for c in range(num_classes):
        mask = gold_labels[:, c] != -1
        if mask.sum() == 0:
            continue
        correct = (predictions[:, c] == gold_labels[:, c]).sum()
        per_class_acc.append(correct / mask.sum())
    return sum(per_class_acc) / len(per_class_acc)

Common pitfalls

• The evaluation uses only 1/10 of the MIMIC-CXR training data and a specific test split (MS-CXR-T); using the full dataset or standard splits will invalidate comparisons.
• Per-disease accuracy is computed as binary classification accuracy for each of the 14 labels, then averaged; treating it as multi-label subset accuracy or macro-F1 will yield different results.
• Report generation metrics (BLEU, ROUGE-L, METEOR) are evaluated on the same test set but require exact text matching against reference radiology reports, not just classification labels.

Evidence (verbatim from paper)

Top-1 accuracy is applied to evaluate the performance of different methods. The best-performing models are highlighted in red, with the second-best in blue.

Citation

@misc{yang2025learningfromall,
  title={Learning from All: Concept Alignment for Autonomous Distillation from Multiple Drifting MLLMs},
  author={Xiaoyu Yang et al. (2025)},
  year={2025},
  note={arXiv:2510.04142}
}

• arXiv: 2510.04142