Rood Mri Eval

rood-mri-eval

ROOD-MRI: Benchmarking the robustness of deep learning segmentation models to out-of-distribution and corrupted data in MRI — Boone et al. (2022) (arXiv:2203.06060, 2022)

What this evaluates

Evaluates the robustness of deep learning segmentation models to out-of-distribution MRI data and synthetic corruptions (noise, contrast, resolution, spatial shifts, motion artifacts) across multiple severity levels. It measures performance degradation on anatomical and lesion segmentation tasks compared to clean data.

Datasets

• ROOD-MRI Benchmark (Hippocampus, Ventricle, WMH) — total 1475; splits: train (1132), test (343); repo https://github.com/AICONSlab/roodmri

Metrics

• DSC (primary) — range: [0, 1]
  • 2|A∩B| / (|A|+|B|), where A and B are prediction and ground truth voxel sets. Also expressed as 2TP/(2TP+FP+TN).
• HD95 — range: mm
  • 95th percentile of the maximum nearest-neighbor distances between prediction and ground truth voxel sets. Null predictions are excluded from calculation.

Input / output format

Input: 3D MRI volumes (T1 or FLAIR sequences) with corresponding binary ground truth segmentation masks. Test images are synthetically corrupted using 11 transforms at 5 severity levels.

Output: Binary 3D segmentation masks predicting the target anatomical structure or lesion.

Scoring recipe

def dice(pred, gt):
    intersection = np.sum(pred & gt)
    return 2.0 * intersection / (np.sum(pred) + np.sum(gt))

def hd95(pred, gt):
    if np.sum(pred) == 0 or np.sum(gt) == 0:
        return np.nan
    dists = []
    for p in np.argwhere(pred):
        dists.append(np.min(np.linalg.norm(p - np.argwhere(gt), axis=1)))
    for g in np.argwhere(gt):
        dists.append(np.min(np.linalg.norm(g - np.argwhere(pred), axis=1)))
    return np.percentile(dists, 95)

Common pitfalls

• Null predictions yield infinite HD95; the paper explicitly excludes them from HD95 calculations and reports the count of null predictions separately.
• Ground truth masks must be transformed and re-discretized for spatial transforms (e.g., affine, elastic deformation, downsampling), not just the images.
• Severity levels are calibrated via visualization rather than quantitative clinical ranges, so they may require re-optimization for different datasets or tasks.

Evidence (verbatim from paper)

We use two metrics to highlight unique characteristics about the similarity between a segmentation prediction and the ground truth. To quantify the overlap, we use the Dice similarity coefficient (DSC). Given two sets A and B (e.g., a binary segmentation prediction and the ground truth), the DSC is defined as... As a distinct but complementary metric, we use the Hausdorff distance to quantify the greatest distance from a point marked as positive in the prediction to the closest point marked as positive in the ground truth, or vice versa. We modified the traditional definition by substituting the 95th percentile instead of the maximum in the max-min operation. The modified metric is often referred to as the modified (95th-percentile) Hausdorff distance (HD95).

Citation

@misc{boone2022roodmri,
  title={ROOD-MRI: Benchmarking the robustness of deep learning segmentation models to out-of-distribution and corrupted data in MRI},
  author={Boone et al. (2022)},
  year={2022},
  note={arXiv:2203.06060}
}

• arXiv: 2203.06060