Histo Vl Eval

histo-vl-eval

How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark — Al Majzoub et al. (2025) (arXiv:2503.12990, 2025)

What this evaluates

Evaluates vision-language foundation models on diverse histopathology clinical tasks (detection, subtyping, grading, mutation prediction) to assess their robustness to textual/visual perturbations, magnification changes, stain normalization, and model calibration.

Datasets

• CRC-100K — total ?; splits: test (-1)
• BreakHist — total ?; splits: test (-1)
• DataBiox — total ?; splits: test (-1)
• GasHisSDB — total ?; splits: test (-1)
• Breast IDC — total ?; splits: test (-1)
• LC25000-lung — total ?; splits: test (-1)

Metrics

• balanced accuracy (primary) — range: [0, 1]
  • Mean of recall scores across all classes. For binary tasks, it equals (TPR + TNR) / 2. It mitigates class imbalance by treating each class equally regardless of sample count.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
• precision — range: [0, 1]
  • Ratio of true positive predictions to all positive predictions: TP / (TP + FP).
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient: (TPTN - FPFN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)). Ranges from -1 to 1, with 1 representing perfect prediction.
• ECE — range: [0, 1]
  • Expected Calibration Error: weighted sum of absolute differences between predicted confidence and actual accuracy across probability bins. Lower values indicate better calibration.

Input / output format

Input: Histopathology image patches extracted from whole slide images, paired with a single or ensemble textual caption/prompt describing the tissue, magnification, or class.

Output: Predicted class labels or probability distributions for tasks including binary detection, multi-class subtyping, grading, and mutation prediction.

Scoring recipe

def balanced_accuracy(y_true, y_pred):
    recalls = [np.mean(y_true[y==c] == c) for c in np.unique(y_true)]
    return np.mean(recalls)

def ece(y_true, y_prob, n_bins=15):
    confidences = np.max(y_prob, axis=1)
    predictions = np.argmax(y_prob, axis=1)
    accuracies = (predictions == y_true).astype(float)
    bin_boundaries = np.linspace(0, 1, n_bins + 1)
    ece = 0.0
    for i in range(n_bins):
        mask = (confidences > bin_boundaries[i]) & (confidences <= bin_boundaries[i+1])
        if np.sum(mask) > 0:
            bin_acc = np.mean(accuracies[mask])
            bin_conf = np.mean(confidences[mask])
            ece += np.sum(mask) / len(y_true) * abs(bin_acc - bin_conf)
    return ece

Common pitfalls

• Models exhibit high sensitivity to prompt/caption variations, with balanced accuracy fluctuating by up to 26% across different textual descriptions of the same image.
• High balanced accuracy does not imply reliable uncertainty estimation; models consistently show high ECE and low confidence, indicating severe miscalibration for clinical deployment.
• Performance is heavily confounded by preprocessing choices, particularly stain normalization and magnification level, which vary across datasets and affect cellular vs. tissue-level information.

Evidence (verbatim from paper)

As per Figure 5(a), all models exhibit high ECE values across tasks. A general trend of the highest ECE values in tissue phenotyping, followed by TIL detection, MSI detection, and cancer grading is observed.

Citation

@misc{almajzoub2025histovl,
  title={How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark},
  author={Al Majzoub et al. (2025)},
  year={2025},
  note={arXiv:2503.12990}
}

• arXiv: 2503.12990