Champkit Eval

champkit-eval

Evaluating histopathology transfer learning with ChampKit — Kaczmarzyk et al. (2022) (arXiv:2206.06862, 2022)

What this evaluates

Evaluates the transfer learning capability and generalization of deep learning models (CNNs and ViTs) on patch-level histopathology image classification tasks across multiple cancer-related benchmarks.

Datasets

• Various publicly available histopathology datasets — total ?; splits: test (-1); repo https://github.com/kaczmarj/champkit

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall, computed at a fixed decision threshold of 0.5.

Input / output format

Input: 224x224 RGB histopathology image patches, normalized to ImageNet statistics or mean/std of 0.5 depending on the pretraining regime.

Output: Class probabilities or logits for patch-level classification, thresholded at 0.5 for F1 calculation.

Scoring recipe

def compute_metrics(predictions, labels):
    threshold = 0.5
    preds_binary = (predictions > threshold).astype(int)
    f1 = f1_score(labels, preds_binary, average='macro')
    auroc = roc_auc_score(labels, predictions)
    return auroc, f1

Common pitfalls

• Using inconsistent hyperparameters across models or datasets invalidates the fair comparison the framework aims to provide.
• Evaluating generalization on a single dataset rather than across multiple tasks masks poor cross-task transfer performance.
• Mismatching image normalization (ImageNet vs. 0.5) between pretraining and fine-tuning can degrade performance.

Evidence (verbatim from paper)

Area under the receiver operating characteristic curve (AUROC) and F1-score (threshold=0.5) were calculated (using torchmetrics [66].

Citation

@misc{kaczmarzyk2022champkit,
  title={Evaluating histopathology transfer learning with ChampKit},
  author={Kaczmarzyk et al. (2022)},
  year={2022},
  note={arXiv:2206.06862}
}

• arXiv: 2206.06862