Hibou Pathology Eval

hibou-pathology-eval

Hibou: A Family of Foundational Vision Transformers for Pathology — Nechaev et al. (2024) (arXiv:2406.05074, 2024)

What this evaluates

Evaluates the generalization and classification capabilities of foundational vision transformers on histopathology data across patch-level tissue classification, slide-level cancer subtyping, and nuclei segmentation tasks.

Datasets

• CRC-100K — total 107180; splits: train (-1), val (-1), test (-1)
• MHIST — total 3152; splits: train (-1), val (-1), test (-1)
• PCam — total 327680; splits: train (-1), val (-1), test (-1)
• MSI-CRC — total 193312; splits: train (-1), val (-1), test (-1)
• MSI-STAD — total 218578; splits: train (-1), val (-1), test (-1)
• TIL-DET — total 304097; splits: train (-1), val (-1), test (-1)
• BRCA — total 963; splits: train (-1), val (-1), test (-1)
• NSCLC — total 973; splits: train (-1), val (-1), test (-1)
• RCC — total 927; splits: train (-1), val (-1), test (-1)
• PanNuke — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• top-1 accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified patches out of total test patches.
• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve for slide-level binary or multi-class classification.
• Average PQ — range: [0, 1]
  • Average Panoptic Quality across three PanNuke splits for nuclei instance segmentation.
• Precision, Recall, F1 — range: [0, 1]
  • Standard instance segmentation metrics averaged across three PanNuke splits for each nuclear category.

Input / output format

Input: H&E-stained histopathology images (patches of 96x96 to 224x224 pixels) or whole slide images (WSIs) divided into non-overlapping foreground patches.

Output: Class labels for patches and slides; segmentation masks and nuclear class labels for segmentation.

Scoring recipe

def evaluate_patch_level(model, dataloader):
    features = model.extract_features(dataloader)
    linear_layer = train_linear_layer(features, labels, optimizer='SGD', lr_schedule='cosine', augmentations=None)
    best_checkpoint = select_best_val_checkpoint(linear_layer)
    test_preds = best_checkpoint.predict(dataloader.test)
    return accuracy(test_preds, dataloader.test.labels)

def evaluate_slide_level(model, wsi_patches):
    patch_features = model.extract_features(wsi_patches)
    pooling_model = train_attention_pooling(patch_features, wsi_labels, optimizer='AdamW', augmentations=None)
    best_checkpoint = select_best_val_checkpoint(pooling_model)
    test_preds = best_checkpoint.predict(wsi_patches.test)
    return auc(test_preds, wsi_patches.test.labels)

Common pitfalls

• Linear probing training uses no data augmentations.
• Slide-level evaluation freezes the pretrained feature extractor and only updates the attention pooling layer parameters.
• Segmentation metrics are averaged over three different PanNuke splits following the CellViT protocol, not a single split.

Evidence (verbatim from paper)

To evaluate our models we use public datasets and perform evaluation on both patch-level and slide-level tasks. We use a linear probing protocol. We extract features from each image using the pretrained model and then train a linear layer to perform classification. Table 1: Linear probing benchmarks reporting top-1 accuracy. Table 2: AUC, WSI subtyping benchmarks, test subset Table 3: Average PQ across the three PanNuke splits for each nuclear category.

Citation

@misc{nechaev2024hibou,
  title={Hibou: A Family of Foundational Vision Transformers for Pathology},
  author={Nechaev et al. (2024)},
  year={2024},
  note={arXiv:2406.05074}
}

• arXiv: 2406.05074