Chexficient Eval

chexficient-eval

A data- and compute-efficient chest X-ray foundation model beyond aggressive scaling — Chong Wang et al. (2026) (arXiv:2602.22843, 2026)

What this evaluates

Evaluates a chest X-ray vision-language foundation model across zero-shot classification, cross-modal retrieval, and adapted downstream tasks (classification, segmentation, report generation). It specifically probes data and compute efficiency, as well as the model's ability to represent long-tailed thoracic diseases without aggressive scaling.

Datasets

• SIIM-PTX — total 1372; splits: test (1372)
• Pneumonia2017 — total 624; splits: test (624)
• TBX11K — total 1800; splits: test (1800)
• CheXpert — total 500; splits: test (500)
• MIMIC-CXR — total 3082; splits: test (3082)
• ChestX-ray14 — total 25596; splits: test (25596)
• VinDr-CXR — total 3000; splits: test (3000)
• VinDr-PCXR — total 1397; splits: test (1397)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve; measures the probability that a randomly chosen positive instance is ranked higher than a negative instance. Reported as macro-average for multi-label tasks.
• Recall@1 — range: [0, 1]
  • Fraction of queries where the correct paired image or report is retrieved within the top-1 result.
• Dice score — range: [0, 1]
  • 2 * |A ∩ B| / (|A| + |B|), measuring overlap between predicted and ground-truth segmentation masks.
• RadGraph — range: percent
  • Standard radiology report generation metric that evaluates clinical concept matching between generated and reference reports.

Input / output format

Input: Chest X-ray images paired with radiology reports. For zero-shot tasks: image + class description prompt (classification) or image/report as query (retrieval). For downstream tasks: image for classification/segmentation; image + reference report for generation.

Output: Classification labels (binary or multi-label), retrieved report/image, segmentation mask, or generated radiology report.

Scoring recipe

def compute_metrics(preds, gold):
    # AUROC (macro-average for multi-label)
    auroc_scores = [roc_auc_score(gold[:, i], preds[:, i]) for i in range(num_classes)]
    auroc = np.mean(auroc_scores)
    # Recall@1
    recall_at_1 = np.mean([1 if p == g else 0 for p, g in zip(preds, gold)])
    # Dice
    dice = 2 * np.sum(pred_mask * gold_mask) / (np.sum(pred_mask) + np.sum(gold_mask))
    return {'AUROC': auroc, 'Recall@1': recall_at_1, 'Dice': dice}

Common pitfalls

• Confusing zero-shot evaluation (no weight updates) with adapted downstream fine-tuning (linear probing or head fine-tuning).
• Reporting single-label AUROC instead of macro-averaged AUROC for multi-disease benchmarks like CheXpert or VinDr-CXR.
• Combining Findings and Impressions sections for cross-modal retrieval without reporting section-specific retrieval performance.

Evidence (verbatim from paper)

The area under the receiver operating curve (AUROC), Recall@1, Dice score, and standard radiology report generation metrics (e.g., RadGraph) are utilized for evaluating the task performance of these models.

Citation

@misc{wang2026chexficient,
  title={A data- and compute-efficient chest X-ray foundation model beyond aggressive scaling},
  author={Chong Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.22843}
}

• arXiv: 2602.22843