Ct Rate Zero Shot Eval

ct-rate-zero-shot-eval

Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography — Hamamci et al. (2024) (arXiv:2403.17834, 2024)

What this evaluates

This benchmark evaluates the zero-shot multi-abnormality detection capability of a visual-language foundation model on 3D chest CT volumes. It probes the model's ability to generalize to unseen data distributions and classify multiple pathologies simultaneously without task-specific supervised training.

Datasets

• CT-RATE — total 50188; splits: train (-1), val (-1); repo https://github.com/ibrahimethemhamamci/CT-CLIP
• RAD-ChestCT — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, computed per abnormality and averaged across all 18 abnormalities.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall, computed per abnormality and averaged.
• accuracy — range: [0, 1]
  • Proportion of correctly classified instances per abnormality, averaged across abnormalities.
• precision — range: [0, 1]
  • Proportion of true positive predictions among all positive predictions per abnormality, averaged.

Input / output format

Input: 3D chest CT volume (reconstructed from DICOM slices)

Output: Softmax probabilities for each of the 18 predefined abnormalities, indicating the likelihood of presence.

Scoring recipe

for each abnormality in 18:
    pos_logit = cosine_similarity(volume_emb, positive_prompt_emb)
    neg_logit = cosine_similarity(volume_emb, negative_prompt_emb)
    logits = [pos_logit, neg_logit]
    probs = softmax(logits)
    pred_prob = probs[0]  # positive logit probability
    pred_label = 1 if pred_prob > threshold else 0
    compute AUROC, F1, accuracy, precision against ground truth
average metrics across all 18 abnormalities

Common pitfalls

• Treating the task as single-label classification instead of multi-label, as medical CTs often contain multiple co-occurring abnormalities.
• Ignoring prompt engineering variations, which significantly impact zero-shot classification performance.
• Failing to account for distribution shift by only evaluating on the internal validation set without the external RAD-ChestCT set.

Evidence (verbatim from paper)

Our zero-shot approach yields significantly higher mean AUROC (0.099 higher in internal validation and 0.082 higher in external validation), mean F1 score (0.061 higher in internal validation and 0.052 higher in external validation), mean accuracy (0.07 higher in internal validation and 0.052 higher in external validation), and mean precision (0.065 higher in internal validation and 0.047 higher in external validation) compared to the supervised baseline.

Citation

@misc{hamamci2024developing,
  title={Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography},
  author={Hamamci et al. (2024)},
  year={2024},
  note={arXiv:2403.17834}
}

• arXiv: 2403.17834