Camchex Eval

camchex-eval

Clinically-aligned Multi-modal Chest X-ray Classification — Sloan et al. (2025) (arXiv:2511.09581, 2025)

What this evaluates

Evaluates a multimodal framework's ability to classify thoracic diseases at the study level by jointly modeling multi-view chest X-rays, clinical indications, and vital signs. It probes the model's capacity to integrate heterogeneous clinical data for accurate multi-label diagnosis across head, body, and tail disease categories.

Datasets

• MIMIC-CXR — total 377110; splits: test (-1)
• CXR-LT 2023 — total ?; splits: test (-1)
• CXR-LT 2024 — total ?; splits: test (-1)

Metrics

• mAP (primary) — range: [0, 1]
  • Mean Average Precision: computes the average of the Average Precision scores across all classes, averaging over varying decision thresholds. It measures performance across thresholds and does not degrade under class imbalance.
• AUROC (macro) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve (macro): calculates the AUC for each class independently and averages them. Can be disproportionately influenced by long-tailed class imbalance.

Input / output format

Input: Per study: frontal and lateral chest X-ray images (1024×1024), clinical indication text, and structured vital signs (temperature, heart rate, respiration rate, blood pressure).

Output: Multi-label probability vector over 14 categories (MIMIC-CXR) or 26 categories (CXR-LT) representing common thoracic diseases and findings.

Scoring recipe

def compute_mAP(y_true, y_pred, num_classes):
    aps = [average_precision_score(y_true[:, c], y_pred[:, c]) for c in range(num_classes)]
    return np.mean(aps)

def compute_AUROC(y_true, y_pred, num_classes):
    aucs = [roc_auc_score(y_true[:, c], y_pred[:, c]) for c in range(num_classes)]
    return np.mean(aucs)

Common pitfalls

• AUROC is heavily skewed by long-tail class imbalance, making mAP the preferred primary metric for CXR-LT benchmarks.
• Evaluation is performed at the study level (aggregating frontal and lateral views), not at the individual image level.
• MIMIC-CXR labels are derived using the CheXpert rule-based extractor, which may introduce noise or differ from manual ground truth.

Evidence (verbatim from paper)

In line with the CXR-LT benchmarks, we evaluate CaMCheX with commonly used metrics, mAP and AUC-ROC (macro). The primary evaluation metric is mAP since AUC-ROC can be disproportionately influenced due to long tailed class imbalance. In contrast, mAP measures performance across decision thresholds and does not degrade under class imbalance.

Citation

@misc{sloan2025camchex,
  title={Clinically-aligned Multi-modal Chest X-ray Classification},
  author={Sloan et al. (2025)},
  year={2025},
  note={arXiv:2511.09581}
}

• arXiv: 2511.09581