Ssl Pathology Benchmark Eval

ssl-pathology-benchmark-eval

Benchmarking Self-Supervised Learning on Diverse Pathology Datasets — Kang et al. (2022) (arXiv:2212.04690, 2022)

What this evaluates

Evaluates the transfer learning capability of self-supervised learning (SSL) pre-trained models on diverse histopathology datasets. It probes domain-specific representation learning by measuring performance on image classification and nuclei instance segmentation tasks under linear probing and fine-tuning protocols.

Datasets

• BACH — total ?; splits: (unstated)
• CRC — total ?; splits: (unstated)
• PCam — total ?; splits: (unstated)
• MHIST — total ?; splits: (unstated)
• CoNSeP — total ?; splits: (unstated)

Metrics

• top-1 accuracy (primary) — range: [0, 1]
  • Fraction of correctly classified instances out of the total number of instances in the test set.
• panoptic quality (PQ) — range: [0, 1]
  • PQ = (Σ IoU(p,g) for TP pairs) / (|TP| + 0.5|FP| + 0.5|FN|). Pairs with IoU > 0.5 are counted as True Positives. Multi-class PQ (mPQ) is used to measure performance simultaneously across classes.

Input / output format

Input: Image patches extracted from whole slide images (WSIs) at various resolutions (e.g., 2048×1536, 224×224, 96×96, 1000×1000 pixels), labeled with cancer type or nuclei instance masks.

Output: For classification: predicted class label. For segmentation: predicted instance masks with class labels.

Scoring recipe

# Classification
accuracy = sum(pred == gold for pred, gold in zip(predictions, gold_labels)) / len(gold_labels)

# Segmentation (PQ)
tp_iou_sum = 0
tp_count = 0
fp_count = 0
for pred in predictions:
    matched = False
    for gold in gold_masks:
        iou = intersection(pred, gold) / union(pred, gold)
        if iou > 0.5:
            tp_iou_sum += iou
            tp_count += 1
            matched = True
            break
    if not matched:
        fp_count += 1
fn_count = len(gold_masks) - tp_count
pq = tp_iou_sum / (tp_count + 0.5 * fp_count + 0.5 * fn_count)

Common pitfalls

• The paper uses 'ImageNet epoch' for training duration, which differs from standard epoch counting based on dataset size.
• Pre-training is primarily conducted on TCGA only, despite mentioning TULIP, which may limit claims about generalization across all sources.
• PQ calculation strictly counts pairs with IoU > 0.5 as TP, a convention that differs from some other instance segmentation metrics.

Evidence (verbatim from paper)

For image classification, we report top-1 accuracy, while using panoptic quality (PQ) [38] for nuclei instance segmentation. PQ is a standard metric for assessing the performance of nuclear instance segmentation [29] that accounts for both detection and segmentation quality with respect to each instance. The PQ metric is defined as, $$ PQ = \frac {\sum_ {(p , g) \in T P} \operatorname {I o U} (p , g)}{| T P | + \frac {1}{2} | F P | + \frac {1}{2} | F N |}, $$ where p denotes a predicted mask for each nuclei class and g denotes a corresponding ground truth.

Citation

@misc{kang2022benchmarking,
  title={Benchmarking Self-Supervised Learning on Diverse Pathology Datasets},
  author={Kang et al. (2022)},
  year={2022},
  note={arXiv:2212.04690}
}

• arXiv: 2212.04690