Opensrh Classification Eval

opensrh-classification-eval

OpenSRH: optimizing brain tumor surgery using intraoperative stimulated Raman histology — Jiang et al. (2022) (arXiv:2206.08439, 2022)

What this evaluates

Evaluates deep learning models on patch-based and patient-level multiclass classification of brain tumor histology images. It probes the ability of CNNs and vision transformers to distinguish between different tumor types and normal tissue using intraoperative stimulated Raman histology data.

Datasets

• OpenSRH — total 1300; splits: train (-1), val (-1), test (-1)

Metrics

• top-1 accuracy (primary) — range: percent
  • Percentage of correctly classified patches or patients out of the total number of instances.
• mean class accuracy (MCA) — range: percent
  • Average of per-class accuracy scores, computed as the mean of (true positives / total actual positives) across all classes.
• mean average precision (MAP) — range: [0, 1]
  • Mean of the average precision scores across all classes, measuring the area under the precision-recall curve for each class.

Input / output format

Input: 300×300 pixel images (ResNet50) or 224×224 pixel images (ViT-S) representing histology patches, with optional horizontal/vertical flipping augmentations.

Output: Patch-level logits processed through softmax, then aggregated via average pooling to produce slide- or patient-level class probabilities.

Scoring recipe

# Aggregate patch-level softmax probabilities to patient/slide level
patient_probs = np.mean(patch_softmax_probs, axis=0)
patient_pred = np.argmax(patient_probs)
# Compute metrics
accuracy = np.mean(patient_pred == patient_labels)
mca = np.mean([np.mean(patient_pred[patient_labels==c] == c) for c in classes])
map_score = average_precision_score(patient_labels, patient_probs, average='macro')

Common pitfalls

• Confusing patch-level metrics with patient-level metrics, as aggregation via average pooling significantly changes performance scores.
• Assuming Vision Transformers outperform CNNs without large-scale pretraining, as ViT-S underperforms ResNet50 on this dataset due to low inductive bias.
• Overlooking the known diagnostic difficulty between HGG and LGG, which causes consistent false negatives across both models and pathologists.

Evidence (verbatim from paper)

Model performance evaluation metrics include top-1 accuracy, mean class accuracy (MCA), and mean average precision (MAP). We aggregated patch-level logits after softmax using average pooling to compute slide and patient-level prediction. We preferred using average pooling over hard patch voting to retain the full patch-level model predictions during slide- or patient-level inference.

Citation

@misc{jiang2022opensrh,
  title={OpenSRH: optimizing brain tumor surgery using intraoperative stimulated Raman histology},
  author={Jiang et al. (2022)},
  year={2022},
  note={arXiv:2206.08439}
}

• arXiv: 2206.08439