Cxmind Chest Xray Eval

cxmind-chest-xray-eval

CX-Mind: A Pioneering Multimodal Large Language Model for Interleaved Reasoning in Chest X-ray via Curriculum-Guided Reinforcement Learning — Li et al. (2025) (arXiv:2508.03733, 2025)

What this evaluates

Evaluates multimodal large language models on chest X-ray diagnosis across visual understanding, text generation, spatiotemporal alignment, and foundational medical language capabilities. It probes the model's ability to interpret radiological images, generate clinical reports, localize anomalies, and reason over medical text.

Datasets

• MIMIC-CXR & CheXpert — total ?; splits: train (-1), test (2000)
• OpenI — total ?; splits: test (500)
• Language Datasets (CHIP-CDN, CMeEE, IMCS-V2-MRG, DDx-basic, MedSafety, MedHG, Med-Exam) — total ?; splits: train (-1), test (-1)
• MS-CXR, RSNA, CXR-AL14 — total ?; splits: test (-1)

Metrics

• Accuracy (Acc) (primary) — range: [0, 1]
  • Proportion of correct predictions: sum(pred == gold) / total samples.
• Jaccard Index — range: [0, 1]
  • Intersection over union of predicted and gold sets: |A ∩ B| / |A ∪ B|. A threshold of >0.5 is applied for task success.
• BLEU — range: [0, 1]
  • Standard corpus-level n-gram precision with brevity penalty for text generation.
• ROUGE-1/2/L — range: [0, 1]
  • Recall-based overlap of unigrams, bigrams, and longest common subsequences between generated and reference text.
• BERTScore — range: [0, 1]
  • Cosine similarity between contextual embeddings of generated and reference tokens, averaged across the sequence.
• Intersection over Union (IoU) — range: [0, 1]
  • Area of overlap between predicted and ground-truth bounding boxes divided by their union area. A threshold of >0.5 is used.

Input / output format

Input: Chest X-ray image(s) paired with a text prompt or question (e.g., disease identification, report generation, view classification, or medical QA).

Output: Text response containing disease labels, a list of identified conditions, a full radiology report, or bounding box coordinates/localization ranges.

Scoring recipe

def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    jaccard = [len(set(p) & set(g)) / len(set(p) | set(g)) for p, g in zip(predictions, golds)]
    iou = [compute_bbox_iou(p, g) for p, g in zip(predictions, golds)]
    bleu = nltk.bleu_score.corpus_bleu([[g] for g in golds], predictions)
    rouge = nltk.rouge_score.corpus_rouge(golds, predictions)
    bertscore = bert_score.score(predictions, golds)
    return {'acc': acc, 'jaccard': jaccard, 'iou': iou, 'bleu': bleu, 'rouge': rouge, 'bertscore': bertscore}

Common pitfalls

• Open-ended disease identification requires post-processing with an LLM (ChatGPT-4o) to extract standardized answers before scoring.
• Jaccard and IoU metrics use a strict threshold (>0.5) to determine task success or filter results.
• Out-of-domain evaluation uses a separate, smaller test set (OpenI) with different disease category coverage than in-domain data.

Evidence (verbatim from paper)

For close-ended questions, accuracy (Acc) was used as the primary metric. For open-ended questions, disease identification tasks were assessed using the Jaccard index and accuracy, with a threshold of Jaccard $>0.5$. For report generation tasks, we adopted standard natural language generation metrics, including BLEU, ROUGE-1, ROUGE-2, and ROUGE-L. Additionally, BERTScore was utilized to evaluate semantic similarity.

Citation

@misc{li2025cxmind,
  title={CX-Mind: A Pioneering Multimodal Large Language Model for Interleaved Reasoning in Chest X-ray via Curriculum-Guided Reinforcement Learning},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2508.03733}
}

• arXiv: 2508.03733