Multi Anatomy Xray Eval

multi-anatomy-xray-eval

Multi Anatomy X-Ray Foundation Model — Singla et al. (2025) (arXiv:2509.12146, 2025)

What this evaluates

Evaluates the generalization and task-specific performance of a multi-anatomy X-ray foundation model across diverse downstream tasks including image retrieval, disease classification, anatomical segmentation, lesion localization, and clinical report generation.

Datasets

• CheXpert — total ?; splits: test (-1)
• dXR — total ?; splits: test (-1)
• PTX / SIIM-ACR — total ?; splits: test (-1)
• MURA — total ?; splits: test (-1)
• JSRT — total ?; splits: test (-1)
• VinDr-RibCXR — total ?; splits: test (-1)
• PAX-Ray++ — total ?; splits: test (-1)
• MS-CXR — total ?; splits: test (-1)
• IU-XRay — total ?; splits: test (-1)
• Bone Fracture Detection — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance across all classification thresholds.
• DSC — range: [0, 1]
  • Dice Similarity Coefficient. Computed as 2 * |A ∩ B| / (|A| + |B|), where A and B are the predicted and ground truth segmentation masks. Measures spatial overlap between masks.
• mIoU — range: [0, 1]
  • Mean Intersection over Union. Average of IoU scores across all classes or instances. IoU = |A ∩ B| / |A ∪ B|.
• P@k — range: [0, 1]
  • Precision at k. Fraction of relevant images among the top-k retrieved results. P@k = |relevant ∩ top_k| / k.
• BLEU-1 to BLEU-4, ROUGE-L, CIDEr — range: percent
  • Standard NLP metrics for text generation. BLEU measures n-gram precision with brevity penalty. ROUGE-L uses longest common subsequence. CIDEr uses TF-IDF weighted n-gram similarity against reference reports.

Input / output format

Input: Single X-ray radiograph image (typically 518×518, 1022×1022, or 1540×1540 pixels). For supervised evaluation, task-specific annotations are provided: class labels (classification), pixel-wise masks (segmentation), bounding boxes/keypoints (localization), or paired clinical reports (generation).

Output: Task-dependent: ranked list of retrieved images (retrieval), class probabilities or discrete labels (classification), binary/semantic segmentation masks (segmentation), bounding box coordinates or keypoint heatmaps (localization), or free-text clinical radiology reports (generation).

Scoring recipe

def compute_metrics(predictions, gold, task):
    if task == 'classification':
        return roc_auc_score(gold, predictions)
    elif task == 'segmentation':
        intersection = np.logical_and(predictions, gold).sum()
        union = np.logical_or(predictions, gold).sum()
        return 2 * intersection / (union + 1e-6)
    elif task == 'retrieval':
        k = 5
        relevant = [i for i, p in enumerate(predictions) if p in gold]
        return len([r for r in relevant if r < k]) / k
    elif task == 'localization':
        # mIoU or mAP@50 computed via standard detection/grounding eval
        return compute_mIoU(predictions, gold)
    elif task == 'generation':
        return compute_nlp_metrics(predictions, gold) # BLEU/ROUGE/CIDEr
    return None

Common pitfalls

• Using the [CLS] token for classification yields suboptimal performance for tasks requiring spatial features (e.g., lead markers); patch embeddings should be used instead.
• Decoder complexity matters: simple linear decoders work well for scratch-trained models, while complex decoders like UPerNet require warm-start initialization or larger pretraining data to avoid underperformance.
• Evaluation is highly resolution-dependent; increasing input size from 518 to 1540 pixels significantly boosts detection of subtle pathologies like pneumothorax, but requires reduced batch sizes to fit GPU memory.

Evidence (verbatim from paper)

We benchmark model performance on public and internal datasets, including CheXpert, PTX, MURA and QC. ... Notably, CXR-0 achieves the highest AUROC on CheXpert with full data (85.3), while RadDINO leads on PTX (81.8).

Citation

@misc{singla2025multi,
  title={Multi Anatomy X-Ray Foundation Model},
  author={Singla et al. (2025)},
  year={2025},
  note={arXiv:2509.12146}
}

• arXiv: 2509.12146