Midog Eval

midog-eval

Mitosis domain generalization in histopathology images -- The MIDOG challenge — Aubreville et al. (2022) (arXiv:2204.03742, 2022)

What this evaluates

Evaluates deep learning models for detecting mitotic figures in histopathology whole-slide images, specifically probing their ability to generalize across different scanner-induced domain shifts such as color distribution, contrast, and depth-of-field variations.

Datasets

• MIDOG — total ?; splits: train (-1), test (-1); repo https://github.com/DeepPathology/MIDOG_evaluation_docker

Metrics

• F_1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: F_1 = 2 * (precision * recall) / (precision + recall). Precision is the ratio of true positive detections to all positive predictions, and recall is the ratio of true positives to all ground truth mitotic figures.

Input / output format

Input: Whole-slide histopathology images (WSIs) from various scanners.

Output: Predicted bounding boxes or coordinates for mitotic figures.

Scoring recipe

tp = count of predicted boxes overlapping ground truth mitoses (IoU > threshold)
fp = count of predicted boxes with no matching ground truth
fn = count of ground truth mitoses with no matching prediction
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return f1

Common pitfalls

• Scanner-specific optical variations (color distribution, contrast, depth-of-field) cause severe domain shift, making models trained on one scanner fail on others.
• Borderline or atypical mitotic figures (e.g., late telophase, unclear boundaries) are often inconsistently labeled by experts, leading to ambiguous ground truth and high false-negative rates.
• Ensembling or test-time augmentation significantly boosts performance but may not reflect single-model generalization capability.

Evidence (verbatim from paper)

The domain-adversarial reference method yielded a competitive $F_{1}$ score of 0.718 on the test set and was outperformed by only four approaches (see Table 2). With an $F_{1}$ score of 0.748, the overall best performance was reached by Yang et al., utilizing segmentation and Fourier-domain mixing as augmentation.

Citation

@misc{aubreville2022midog,
  title={Mitosis domain generalization in histopathology images -- The MIDOG challenge},
  author={Aubreville et al. (2022)},
  year={2022},
  note={arXiv:2204.03742}
}

• arXiv: 2204.03742