Unet Biomedical Seg Eval

unet-biomedical-seg-eval

U-Net: Convolutional Networks for Biomedical Image Segmentation — Ronneberger et al. (2015) (arXiv:1505.04597, 2015)

What this evaluates

Evaluates pixel-level biomedical image segmentation capability using convolutional networks. Probes the model's ability to precisely delineate cellular structures and membranes in electron and light microscopy images with limited training data.

Datasets

• EM segmentation challenge (ISBI 2012) — total ?; splits: train (30), test (-1)
• PhC-U373 — total ?; splits: train (35)
• DIC-HeLa — total ?; splits: train (20)

Metrics

• warping error (primary) — range: [0, 1]
  • Computed by thresholding the predicted membrane probability map at 10 different levels and measuring the deformation required to align predictions with ground truth.
• Rand error — range: [0, 1]
  • Measures segmentation discrepancy based on Rand index statistics between predicted and ground truth masks.
• pixel error — range: [0, 1]
  • Standard pixel-wise classification error between predicted and ground truth masks.
• IOU (primary) — range: [0, 1]
  • Intersection over Union, calculated as the area of overlap between the predicted segmentation mask and the ground truth divided by the area of union. Average IOU is reported across the dataset.

Input / output format

Input: Grayscale microscopy images (512x512 pixels for EM; phase contrast or DIC for cell tracking).

Output: Predicted membrane probability map (for EM) or pixel-wise segmentation mask (for cell tracking).

Scoring recipe

def score_em(pred_map, gt_map):
    errors = []
    for thresh in np.linspace(0, 1, 10):
        pred_bin = (pred_map > thresh).astype(int)
        errors.append(compute_warping_error(pred_bin, gt_map))
    return min(errors)

def score_iou(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    return intersection / union if union > 0 else 0.0

Common pitfalls

• The EM test set ground truth is kept secret; evaluation requires submitting predictions to the challenge organizers rather than local testing.
• EM metrics are computed by thresholding the continuous probability map at 10 different levels, not on a single fixed threshold.
• Cell tracking training data is only partially annotated, which may require careful loss weighting or threshold selection during evaluation.

Evidence (verbatim from paper)

The evaluation is done by thresholding the map at 10 different levels and computation of the “warping error”, the “Rand error” and the “pixel error”. Here we achieve an average IOU (“intersection over union”) of 92%, which is significantly better than the second best algorithm with 83%.

Citation

@misc{ronneberger2015unet,
  title={U-Net: Convolutional Networks for Biomedical Image Segmentation},
  author={Ronneberger et al. (2015)},
  year={2015},
  note={arXiv:1505.04597}
}

• arXiv: 1505.04597