Aeropath Airway Segmentation Eval

aeropath-airway-segmentation-eval

AeroPath: An airway segmentation benchmark dataset with challenging pathology — Støverud et al. (2023) (arXiv:2311.01138, 2023)

What this evaluates

This benchmark evaluates 3D medical image segmentation models on contrast-enhanced CT scans containing severe airway pathologies. It probes a model's ability to accurately segment complex, distorted bronchial trees and maintain topological completeness down to small airway generations despite anatomical anomalies like tumors and emphysema.

Datasets

• AeroPath — total 27; splits: test (27); repo https://github.com/raidionics/AeroPath

Metrics

• DSC (primary) — range: percent
  • Dice Similarity Coefficient, calculated as 2 * |prediction ∩ ground_truth| / (|prediction| + |ground_truth|). Measures voxel-wise overlap between the predicted airway mask and the annotated ground truth.
• TD — range: percent
  • Tree Detection rate, representing the percentage of correctly detected airway tree length or generations relative to the ground truth.
• BD — range: percent
  • Branch Detection rate, representing the percentage of correctly detected individual airway branches relative to the ground truth.

Input / output format

Input: 3D contrast-enhanced computed tomography (CT) volumes of the chest.

Output: 3D binary segmentation mask indicating airway voxels.

Scoring recipe

def compute_metrics(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    union = np.sum(pred_mask) + np.sum(gt_mask)
    dsc = (2 * intersection / union) * 100 if union > 0 else 0.0
    tp = np.sum((pred_mask == 1) & (gt_mask == 1))
    fp = np.sum((pred_mask == 1) & (gt_mask == 0))
    fn = np.sum((pred_mask == 0) & (gt_mask == 1))
    tn = np.sum((pred_mask == 0) & (gt_mask == 0))
    precision = (tp / (tp + fp)) * 100 if (tp + fp) > 0 else 0.0
    sensitivity = (tp / (tp + fn)) * 100 if (tp + fn) > 0 else 0.0
    specificity = (tn / (tn + fp)) * 100 if (tn + fp) > 0 else 0.0
    return {'DSC': dsc, 'Precision': precision, 'Sensitivity': sensitivity, 'Specificity': specificity}

Common pitfalls

• Ground truth annotations for some cases were partly generated by baseline tools (FAST + BronchiNet), which may artificially inflate DSC and other metrics for those specific scans.
• Post-processing steps (e.g., connecting fragmented branches, removing islands) improve visual/topological continuity but do not substantially change voxel-wise metrics like DSC.
• Models may produce false positives by segmenting adjacent structures with similar CT intensities (e.g., esophagus, lung tissue) if not carefully constrained.

Evidence (verbatim from paper)

The PW method has the best overall DSC (84.98 ± 3.24) of all tested methods. Adding ensemble to the PW model causes a drop in DSC and precision with 0.5 and 4%, respectively. On the other hand, TD and BD increases with 2.4 and 4.1%, respectively.

Citation

@misc{stoverud2023aeropath,
  title={AeroPath: An airway segmentation benchmark dataset with challenging pathology},
  author={Støverud et al. (2023)},
  year={2023},
  note={arXiv:2311.01138}
}

• arXiv: 2311.01138