Clip Cxr Fairness Eval

clip-cxr-fairness-eval

Fairness and Robustness of CLIP-Based Models for Chest X-rays — Sourget et al. (2025) (arXiv:2507.21291, 2025)

What this evaluates

Evaluates zero-shot classification performance of CLIP-based vision-language models on chest X-rays, assessing fairness across demographic subgroups (age, sex, race) and robustness to spurious correlations (presence of chest drains in pneumothorax cases).

Datasets

• MIMIC-CXR — total ?; splits: test (-1); repo https://github.com/TheoSourget/clip_cxr_fairness

Metrics

• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to discriminate between positive and negative classes across all classification thresholds.
• AUPRCadj (primary) — range: other
  • Adjusted Area Under the Precision-Recall Curve. Computed relative to a random classifier baseline; negative values indicate performance worse than random. Accounts for class imbalance in medical imaging datasets.

Input / output format

Input: Chest X-ray images and text prompts for zero-shot classification of 14 radiological findings.

Output: Predicted probability scores for each pathology class.

Scoring recipe

def compute_metrics(predictions, gold, subgroups=None):
    auc = compute_auc(gold, predictions)
    auprc_adj = compute_auprc(gold, predictions) - baseline_random_auprc
    results = {'AUC': auc, 'AUPRCadj': auprc_adj}
    if subgroups:
        for group in subgroups:
            mask = gold['subgroup'] == group
            results[group] = compute_metrics(predictions[mask], gold[mask])
    return results

Common pitfalls

• AUPRCadj can yield negative values when model performance falls below the random classifier baseline, which may be misinterpreted as invalid scores.
• Small subgroup sample sizes (e.g., Asian patients or 18-25 age group) cause high variance and wide confidence intervals, potentially skewing fairness assessments.
• PCA visualizations of embeddings may appear to show no bias, while linear probing reveals sensitive attributes are actually encoded in the feature space.

Evidence (verbatim from paper)

Table[1] shows the AUC and AUPRCadj of the different models on the MIMIC-CXR test set. While the results vary across the models and subgroups, we can still see a similar pattern with gaps across patient ages. Fig.[4(a)] and [4(b)] show the results on chest X-ray with and without chest drains. We can see that all models except CXR-CLIP obtain better adjusted AUPRC on images with chest drains compared to X-rays without drains (ranging from +0.09 to +0.30)

Citation

@misc{sourget2025fairnessrobustnessclip,
  title={Fairness and Robustness of CLIP-Based Models for Chest X-rays},
  author={Sourget et al. (2025)},
  year={2025},
  note={arXiv:2507.21291}
}

• arXiv: 2507.21291