Smile Uhura Eval

smile-uhura-eval

SMILE-UHURA Challenge -- Small Vessel Segmentation at Mesoscopic Scale from Ultra-High Resolution 7T Magnetic Resonance Angiograms — Chatterjee et al. (2024) (arXiv:2411.09593, 2024)

What this evaluates

Evaluates 3D medical image segmentation models on mesoscopic small vessel extraction from ultra-high-resolution (7T) Time-of-Flight Magnetic Resonance Angiography. It probes a model's ability to handle high noise, poor vessel-background contrast, and domain shifts across different MRI acquisition sources.

Datasets

• SMILE-UHURA Challenge Dataset — total ?; splits: test_open (-1), test_secret (-1)

Metrics

• Dice coefficient (DICE) (primary) — range: [0, 1]
  • 2 * |A ∩ B| / (|A| + |B|), where A and B are the predicted and ground truth binary masks. Ranges from 0 to 1, with 1 indicating perfect voxel-wise overlap.

Input / output format

Input: 3D Time-of-Flight Magnetic Resonance Angiography (MRA) volumes acquired at 7T ultra-high field strength.

Output: 3D binary segmentation mask indicating vessel voxels (foreground) versus background.

Scoring recipe

def compute_dice(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    return 2.0 * intersection / (np.sum(pred_mask) + np.sum(gt_mask))

# Protocol:
# 1. Compute Dice per 3D volume.
# 2. Aggregate across all test volumes.
# 3. Report median ± interquartile range (IQR) for each metric.

Common pitfalls

• bAHD is highly sensitive to extreme boundary outliers; the authors had to truncate visualization plots at bAHD ≤ 3 to maintain readability.
• Mutual Information (MUTINF) showed minimal variance across methods, making it a poor discriminator for segmentation quality in this domain.
• The 'Secret Dataset' intentionally tests generalization to a different acquisition source, so high open-test performance does not guarantee strong secret-test results.

Evidence (verbatim from paper)

The performance of various deep learning methods was evaluated on the open dataset, comprising MRI volumes held out from the training set but sharing identical properties. The metrics assessed included the Dice coefficient (DICE), Jaccard index (JACRD), volumetric similarity coefficient (VOLSMTY), mutual information (MUTINF), and balanced average Hausdorff distance (bAHD, or bAVD according to the EvaluateSegmentation pipeline), each reported as median ± interquartile range (IQR) in Table [3], and shown in Figures [1],[2],[3],[4], and [5].

Citation

@misc{chatterjee2024smileuhura,
  title={SMILE-UHURA Challenge -- Small Vessel Segmentation at Mesoscopic Scale from Ultra-High Resolution 7T Magnetic Resonance Angiograms},
  author={Chatterjee et al. (2024)},
  year={2024},
  note={arXiv:2411.09593}
}

• arXiv: 2411.09593