Pannuke Eval

pannuke-eval

PanNuke Dataset Extension, Insights and Baselines — Gamper et al. (2020) (arXiv:2003.10778, 2020)

What this evaluates

Evaluates the ability of deep learning models to perform simultaneous instance segmentation and nuclear classification on histopathology whole-slide image patches across diverse cancer tissue types.

Datasets

• PanNuke — total 190000; splits: train (-1), val (-1), test (-1)

Metrics

• mPQ (primary) — range: [0, 1]
  • Multi-class Panoptic Quality. Calculated independently for each of the 19 tissue classes and then averaged to equally weight each tissue. True positives require IoU > 0.5. PQ = TP / (TP + 0.5FP + 0.5FN).
• bPQ — range: [0, 1]
  • Binary Panoptic Quality. Treats all nuclei as a single class. Uses the same IoU > 0.5 threshold and PQ formula as mPQ.
• F1 — range: [0, 1]
  • F1 score for nucleus detection. A detection is a true positive if its predicted centroid is within 12 pixels of the ground truth centroid.

Input / output format

Input: Pre-extracted image patches from whole-slide histopathology images.

Output: Pixel-level instance segmentation masks for each nucleus, along with a predicted nuclear class label (e.g., neoplastic, non-neoplastic epithelial, inflammatory, connective, dead).

Scoring recipe

def compute_mpq(predictions, ground_truth):
    pq_scores = []
    for tissue_class in all_tissues:
        pred_masks = [m for m in predictions if m.class == tissue_class]
        gt_masks = [m for m in ground_truth if m.class == tissue_class]
        tp, fp, fn = 0, 0, 0
        for p, g in match_instances(pred_masks, gt_masks, iou_threshold=0.5):
            if p is not None: tp += 1
            else: fp += 1
        fn = len(gt_masks) - tp
        pq_scores.append(tp / (tp + 0.5*fp + 0.5*fn))
    return sum(pq_scores) / len(pq_scores)

def compute_detection_f1(pred_centroids, gt_centroids):
    tp, fp, fn = 0, 0, 0
    for p, g in match_centroids(pred_centroids, gt_centroids, dist_threshold=12):
        if p is not None: tp += 1
        else: fp += 1
    fn = len(gt_centroids) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Small nuclei (e.g., dead cells) struggle to achieve IoU > 0.5, leading to artificially low PQ scores.
• Class imbalance across the 19 tissue types and 5 nuclear categories can skew average metrics if not averaged per-class first.
• Distinguishing neoplastic from non-neoplastic nuclei is inherently ambiguous and often requires contextual information even for expert pathologists.

Evidence (verbatim from paper)

To quantify the instance segmentation performance of each of the models trained on PanNuke, we use panoptic quality (PQ) [2], [45]. ... For mPQ, the PQ is calculated independently for each positive class and then the results are averaged. Therefore, the metric is insensitive to class imbalance. Our main criterion for evaluating model performance is the average mPQ over all of the tissues, which therefore equally weights the contribution of each tissue type. The IoU threshold for determining a true positive during PQ calculation is set to 0.5.

Citation

@misc{gamper2020pannuke,
  title={PanNuke Dataset Extension, Insights and Baselines},
  author={Gamper et al. (2020)},
  year={2020},
  note={arXiv:2003.10778}
}

• arXiv: 2003.10778