# Mimic Cxr Medical Eval

> Evaluates the robustness, consistency, and generalization of a distilled multimodal large language model on medical imaging tasks. It probes the model's ability to perform multi-label chest disease classification and generate diagnostic radiology reports from chest X-ray images, even when trained on limited data from drifting teacher models. Use when the user wants to benchmark on MIMIC-CXR, or asks about evaluating this task. Reports Top-1 accuracy.

- Skill: `qhjqhj00/mimic-cxr-medical-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mimic-cxr-medical-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mimic-cxr-medical-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mimic-cxr-medical-eval

---


# mimic-cxr-medical-eval

> Learning from All: Concept Alignment for Autonomous Distillation from Multiple Drifting MLLMs — Xiaoyu Yang et al. (2025) (arXiv:2510.04142, 2025)

## What this evaluates

Evaluates the robustness, consistency, and generalization of a distilled multimodal large language model on medical imaging tasks. It probes the model's ability to perform multi-label chest disease classification and generate diagnostic radiology reports from chest X-ray images, even when trained on limited data from drifting teacher models.

## Datasets

- **MIMIC-CXR** — total 371920; splits: train (-1), test (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted disease labels out of the total number of test instances. Evaluated as binary classification accuracy for each of the 14 diseases, then averaged across all labels.
- `BLEU-4` — range: [0, 1]
  - N-gram precision metric evaluating the overlap between generated diagnostic reports and reference radiology reports, considering 4-gram matches.
- `ROUGE-L` — range: [0, 1]
  - Recall-oriented metric measuring the longest common subsequence overlap between generated and reference reports to assess narrative completeness.
- `METEOR` — range: [0, 1]
  - Metric that aligns generated and reference text using synonymy and stemming to capture lexical and semantic alignment.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between positive and negative disease labels in zero-shot settings.

## Input / output format

**Input**: Chest X-ray image (and optional instruction prompt for report generation).

**Output**: For classification: predicted disease label(s) or probability scores over 14 categories. For report generation: free-text diagnostic radiology report.

## Scoring recipe

```python
def compute_avg_top1_accuracy(predictions, gold_labels, num_classes=14):
    per_class_acc = []
    for c in range(num_classes):
        mask = gold_labels[:, c] != -1
        if mask.sum() == 0:
            continue
        correct = (predictions[:, c] == gold_labels[:, c]).sum()
        per_class_acc.append(correct / mask.sum())
    return sum(per_class_acc) / len(per_class_acc)
```

## Common pitfalls

- The evaluation uses only 1/10 of the MIMIC-CXR training data and a specific test split (MS-CXR-T); using the full dataset or standard splits will invalidate comparisons.
- Per-disease accuracy is computed as binary classification accuracy for each of the 14 labels, then averaged; treating it as multi-label subset accuracy or macro-F1 will yield different results.
- Report generation metrics (BLEU, ROUGE-L, METEOR) are evaluated on the same test set but require exact text matching against reference radiology reports, not just classification labels.

## Evidence (verbatim from paper)

> Top-1 accuracy is applied to evaluate the performance of different methods. The best-performing models are highlighted in red, with the second-best in blue.

## Citation

```bibtex
@misc{yang2025learningfromall,
  title={Learning from All: Concept Alignment for Autonomous Distillation from Multiple Drifting MLLMs},
  author={Xiaoyu Yang et al. (2025)},
  year={2025},
  note={arXiv:2510.04142}
}
```

- arXiv: 2510.04142