Chex Eval

chex-eval

ChEX: Interactive Localization and Region Description in Chest X-rays — Müller et al. (2024) (arXiv:2404.15770, 2024)

What this evaluates

Evaluates a vision-language model's ability to perform interactive localization, region classification, and text generation on chest X-rays. It probes zero-shot multitask capabilities, including sentence grounding, pathology detection, and customizable report generation.

Datasets

• MS-CXR — total ?; splits: test (-1)
• VinDrCXR — total ?; splits: test (-1)
• NIH8 — total ?; splits: test (-1)
• CIG — total ?; splits: test (-1)
• MIMIC-CXR — total ?; splits: test (-1)

Metrics

• mAP (primary) — range: percent
  • Mean Average Precision across pathology classes, computed by averaging precision-recall curves over different confidence thresholds.
• mIoU — range: [0, 1]
  • Mean Intersection over Union between predicted and ground-truth bounding boxes, averaged across classes.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve for binary region classification.
• wAUROC — range: [0, 1]
  • Weighted AUROC for region classification, accounting for class imbalance.
• Mic-F1-14 — range: percent
  • Micro-averaged F1 score over 14 specific chest pathology findings.
• Mac-F1-14 — range: percent
  • Macro-averaged F1 score over 14 specific chest pathology findings.
• Ex-F1-14 — range: percent
  • Exact-match F1 score for predicting the presence/absence of 14 findings.
• Mic-F1-5+ — range: percent
  • Micro-averaged F1 score over 5+ common findings.
• Mac-F1-5+ — range: percent
  • Macro-averaged F1 score over 5+ common findings.
• METEOR — range: percent
  • METEOR score for evaluating generated text against reference reports, measuring alignment and synonymy.

Input / output format

Input: Chest X-ray image, textual query (e.g., pathology name or regional hint), and optionally a bounding box prompt.

Output: Predicted bounding box coordinates, region classification label, and/or generated text description/sentence.

Scoring recipe

def evaluate(preds, gold):
    scores = {}
    scores['mAP'] = mean_average_precision(preds['boxes'], gold['boxes'])
    scores['mIoU'] = mean_iou(preds['boxes'], gold['boxes'])
    scores['AUROC'] = roc_auc_score(gold['labels'], preds['scores'])
    scores['Mic-F1-14'] = micro_f1(gold['findings'], preds['findings'])
    scores['METEOR'] = meteor_score(gold['text'], preds['text'])
    return scores

Common pitfalls

• Test splits and pre-processing differ across baseline models, complicating direct numerical comparison.
• Baselines are typically specialized for single tasks, making head-to-head comparison with a multitask model inherently uneven.
• Language metrics like METEOR are reported low because the model generates aspect-level sentences rather than full reports.

Evidence (verbatim from paper)

On MS-CXR, ChEX improves by 25% on Mac-F1-14. On CIG, ChEX improves by 18% on Mic-F1-14, 40% on Mac-F1-14, and 29% on METEOR, although RGRG was explicitly trained on this task.

Citation

@misc{muller2024chex,
  title={ChEX: Interactive Localization and Region Description in Chest X-rays},
  author={Müller et al. (2024)},
  year={2024},
  note={arXiv:2404.15770}
}

• arXiv: 2404.15770