Dino V2 Radiology Eval

dino-v2-radiology-eval

Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks — Baharoon et al. (2023) (arXiv:2312.02366, 2023)

What this evaluates

Evaluates the cross-task generalizability of the DINOv2 vision foundation model on medical image analysis tasks, specifically disease classification and organ segmentation across X-ray, CT, and MRI modalities.

Datasets

• NIH Chest X-ray — total ?; splits: test (-1)
• CheXpert — total ?; splits: test (-1)
• SARS-CoV-2 — total ?; splits: test (-1)
• Brain Tumor — total ?; splits: test (-1)
• Montgomery County (MC) — total ?; splits: test (-1)
• AMOS — total ?; splits: test (-1)
• MSD Heart — total ?; splits: test (-1)
• MSD Hipp — total ?; splits: test (-1)
• MSD Spleen — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive and false positive rates across classification thresholds.
• Dice/Jaccard average — range: [0, 1]
  • The arithmetic mean of the Dice coefficient and Jaccard index (Intersection over Union), measuring overlap between predicted and ground-truth segmentation masks.

Input / output format

Input: 2D or 3D medical images (X-ray, CT, MRI) resized to 224×224, 518×518, or 1024×1024 pixels depending on the model and task.

Output: Classification: predicted probabilities or class labels. Segmentation: pixel-wise binary or multi-class masks.

Scoring recipe

def compute_metrics(predictions, gold):
    if is_classification(gold):
        return roc_auc_score(gold, predictions)
    else:
        dice = dice_coefficient(gold, predictions)
        jaccard = jaccard_index(gold, predictions)
        return (dice + jaccard) / 2

Common pitfalls

• kNN evaluation underperforms due to domain shift between natural image pre-training and medical images.
• DINOv2's positional encoding interpolation degrades performance on 224×224 inputs compared to its native 518×518 pre-training size.
• SAM classification requires averaging patch embeddings as a CLS token substitute, which is non-standard.

Evidence (verbatim from paper)

We used the area under the operating receiver curve (AUROC) as a performance metric for classification tasks, and the average of the dice and jaccard scores as a metric for segmentation.

Citation

@misc{baharoon2023dino,
  title={Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks},
  author={Baharoon et al. (2023)},
  year={2023},
  note={arXiv:2312.02366}
}

• arXiv: 2312.02366