Medmnist Linear Probe Eval

medmnist-linear-probe-eval

BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysis — Liu et al. (2025) (arXiv:2505.09329, 2025)

What this evaluates

Evaluates the generalization capability and scaling efficiency of self-supervised vision foundation models on a diverse suite of 12 biomedical image classification tasks. It probes how model capacity, data diversity, and pretraining objectives affect downstream diagnostic performance when using a frozen feature extractor.

Datasets

• MedMNIST (12 benchmarks) — total ?; splits: test (-1)

Metrics

• MCC (primary) — range: [-1, 1]
  • Matthews Correlation Coefficient measures the quality of binary/multiclass classifications. Calculated as (TPTN - FPFN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)). Ranges from -1 to 1, where 1 is perfect prediction.
• BA — range: [0, 1]
  • Balanced Accuracy is the average of recall obtained on each class. Computed as (TP/TP+FP + TN/TN+FN) / 2. It handles class imbalance by giving equal weight to each class.
• F1 — range: [0, 1]
  • F1 score is the harmonic mean of precision and recall. Calculated as 2 * (Precision * Recall) / (Precision + Recall). Typically averaged macro across classes in multi-task settings.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve measures the model's ability to discriminate between classes across all classification thresholds. Computed by integrating the ROC curve.

Input / output format

Input: Biomedical images across various modalities (X-ray, CT, ultrasound, histology, etc.) resized to the model's input resolution.

Output: Predicted class labels (binary or multi-class depending on the specific MedMNIST task).

Scoring recipe

def evaluate_linear_probe(model, images, labels):
    features = model.encode(images)  # Freeze backbone, extract features
    clf = LogisticRegression(max_iter=1000)
    clf.fit(features_train, labels_train)
    preds = clf.predict(features_test)
    mcc = matthews_corrcoef(labels_test, preds)
    ba = balanced_accuracy_score(labels_test, preds)
    f1 = f1_score(labels_test, preds, average='macro')
    auc = roc_auc_score(labels_test, clf.decision_function(features_test), multi_class='ovr')
    return {'MCC': mcc, 'BA': ba, 'F1': f1, 'AUC': auc}

Common pitfalls

• Linear probing evaluates representation quality but may underperform compared to full fine-tuning, which is intentional for benchmarking foundation models.
• Averaging metrics across 12 heterogeneous benchmarks can obscure task-specific scaling plateaus or regressions.
• Results report 95% confidence intervals, but comparisons often rely on point estimates without formal statistical significance testing.

Evidence (verbatim from paper)

As shown in Table [1], BioVFM significantly outperforms existing medical foundation model BiomedGPT [[27]], BiomedCLIP [[29]], and RAD-DINO [[20]] across 12 medical benchmarks with linear-probing by at least 3.32% in MCC, 2.81% in BA, 2.14% in F1 score, and 0.94% in AUC.

Citation

@misc{liu2025biovfm21m,
  title={BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysis},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2505.09329}
}

• arXiv: 2505.09329