Benchx Eval

benchx-eval

BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays — Yang Zhou et al. (2024) (arXiv:2410.21969, 2024)

What this evaluates

Evaluates Medical Vision-Language Pretraining (MedVLP) models on chest X-ray tasks including multi-label/binary classification, segmentation, report generation, and image-text retrieval. It specifically probes how standardized preprocessing and finetuning strategies affect model performance across heterogeneous architectures.

Datasets

• NIH — total ?; splits: train (-1), test (-1)
• VinDr — total ?; splits: train (-1), test (-1)
• COVIDx — total ?; splits: train (-1), test (-1)
• SIIM — total ?; splits: train (-1), test (-1)
• RSNA — total ?; splits: train (-1), test (-1)
• Object-CXR — total ?; splits: train (-1), test (-1)
• TBX11K — total ?; splits: train (-1), test (-1)
• IUXray — total ?; splits: train (-1), test (-1)
• MIMIC 5x200 — total ?; splits: train (-1), test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance across all classification thresholds.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used for binary classification tasks where AUROC may not reflect performance differences accurately.
• mDice — range: [0, 1]
  • Mean Dice coefficient across classes/instances. Measures spatial overlap between predicted and ground truth segmentation masks.
• BLEU/ROUGE-L/METEOR — range: [0, 1]
  • Standard NLG metrics: BLEU measures n-gram precision, ROUGE-L measures longest common subsequence recall, METEOR combines precision, recall, and alignment.
• Hit@K / Precision@K — range: [0, 1]
  • Hit@K (H@K) indicates whether the correct report is in the top K retrieved items. Precision@K (P@K) measures the proportion of correct reports among the top K predictions.

Input / output format

Input: Chest X-ray images. For retrieval tasks, an image query is paired against a candidate pool of radiology reports.

Output: Predicted class labels (multi-label or binary), segmentation masks, generated medical reports, or a ranked list of candidate reports.

Scoring recipe

def score(preds, golds, task):
    if task == 'multilabel':
        return roc_auc_score(golds, preds, average='macro')
    elif task == 'binary':
        preds_bin = (preds >= 0.5).astype(int)
        return f1_score(golds, preds_bin, average='macro')
    elif task == 'seg':
        return 2 * (preds * golds).sum() / (preds.sum() + golds.sum() + 1e-8)
    elif task == 'retrieval':
        ranks = np.argsort(-preds, axis=1)
        correct_idx = (ranks == golds[:, None]).argmax(axis=1)
        return (correct_idx < K).mean(), (correct_idx < K).sum() / K

Common pitfalls

• Using AUROC for binary classification instead of F1, as the paper notes AUROC may not fully reflect performance differences.
• Applying naive finetuning protocols (e.g., simple linear head) without task-specific hyperparameter tuning, which significantly underestimates model capabilities.
• Benchmarking image-text retrieval without accounting for factors beyond image encoders, leading to misleading similarity scores.

Evidence (verbatim from paper)

The evaluation metrics include the area under the ROC curve (AUROC) for multilabel classification, measuring the model’s ability to differentiate between true positives and false positives across various threshold values. For binary classification, we employ F1, the harmonic mean of precision and recall, because we find that AUROC may not fully reflect the performance difference across MedVLP methods.

Citation

@misc{zhou2024benchx,
  title={BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays},
  author={Yang Zhou et al. (2024)},
  year={2024},
  note={arXiv:2410.21969}
}

• arXiv: 2410.21969