Dinov3 Medical Vision Eval

dinov3-medical-vision-eval

Does DINOv3 Set a New Medical Vision Standard? — Liu et al. (2025) (arXiv:2509.06467, 2025)

What this evaluates

Evaluates the cross-domain generalization and scaling behavior of a natural-image pre-trained vision transformer (DINOv3) across diverse medical imaging modalities, including 2D/3D classification and segmentation tasks.

Datasets

• NIH-14 — total ?; splits: test (-1)
• RSNA-Pneumonia — total ?; splits: test (-1)
• Camelyon16 — total ?; splits: train (-1), test (-1)
• Camelyon17 — total ?; splits: test (-1)
• BCNB — total ?; splits: test (-1)
• Kvasir-Capsule — total ?; splits: test (-1)
• AutoLaparo — total ?; splits: test (-1)
• EndoVis18 — total ?; splits: test (-1)
• EDD 2020 — total ?; splits: test (-1)
• CT-RATE — total ?; splits: test (-1)
• Medical Segmentation Decathlon (MSD) — total ?; splits: test (-1)
• CREMI — total ?; splits: test (-1)
• AC3/4 — total ?; splits: test (-1)
• AutoPET-II — total ?; splits: test (-1)
• HECKTOR 2022 — total ?; splits: test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Measures the model's ability to distinguish between classes across all classification thresholds.
• Dice score (primary) — range: [0, 1]
  • 2 * |A ∩ B| / (|A| + |B|), where A and B are the predicted and ground truth masks. Measures overlap between binary segmentation predictions and gold labels.
• IoU (Intersection over Union) (primary) — range: [0, 1]
  • |A ∩ B| / |A ∪ B|. Measures the ratio of the intersection area to the union area between predicted and ground truth masks.
• Accuracy — range: [0, 1]
  • Number of correct predictions divided by total number of predictions.
• F-1 Score — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).
• Jaccard Index — range: [0, 1]
  • Identical to IoU; measures set similarity between predicted and ground truth masks.
• VOI (Variation of Information) — range: [0, ∞)
  • Information-theoretic measure of segmentation error; lower values indicate better agreement between predicted and ground truth segmentations.
• ARAND (Adjusted Rand) — range: [0, 1]
  • Adjusted Rand index measuring similarity between two clusterings; lower error rates indicate better segmentation quality.

Input / output format

Input: Medical images (2D X-rays, pathology patches/WSIs, endoscopic frames, 3D CT/PET volumes) resized or processed to fixed input resolutions (e.g., 256x256 or 512x512).

Output: Class probabilities or logits for classification tasks; pixel-wise segmentation masks for segmentation tasks.

Scoring recipe

def compute_metrics(preds, gold):
    # Classification
    auc = roc_auc_score(gold, preds)
    acc = accuracy_score(gold, preds.argmax(axis=1))
    # Segmentation
    intersection = np.sum(pred_mask & gold_mask)
    dice = 2 * intersection / (np.sum(pred_mask) + np.sum(gold_mask))
    iou = intersection / (np.sum(pred_mask | gold_mask))
    return {'AUC': auc, 'Dice': dice, 'IoU': iou}

Common pitfalls

• Performance does not consistently scale with model size or resolution; larger models or higher resolutions can degrade performance.
• Frozen backbone features often lack fine-grained textural details required for histopathology or electron microscopy segmentation.
• 2D slice-wise feature aggregation may be insufficient for volumetric 3D tasks without 3D-specific adapters or pooling strategies.

Evidence (verbatim from paper)

As shown in Table [2], DINOv3-L achieves the highest AUC on NIH-14, outperforming the medical-specific BiomedCLIP model. While BiomedCLIP performs best on the RSNA-Pneumonia task, DINOv3 models are close contenders. However, the results also highlight an inconsistent scaling behavior, as seen in Figure [1]. Performance does not reliably improve with larger model sizes or higher input resolutions; for instance, AUC for all models on NIH-14 peaks at a 512x512 resolution before declining.

Citation

@misc{liu2025dino3medical,
  title={Does DINOv3 Set a New Medical Vision Standard?},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2509.06467}
}

• arXiv: 2509.06467