Coralscapes Eval

coralscapes-eval

The Coralscapes Dataset: Semantic Scene Understanding in Coral Reefs — Sauder et al. (2025) (arXiv:2503.20000, 2025)

What this evaluates

Probes semantic segmentation models on complex underwater scenes characterized by high morphological variability, degradation states, and visual distortions. It evaluates the model's ability to generalize across geographically distinct reef sites and handle severe class imbalance and fine-grained benthic classification.

Datasets

• Coralscapes — total 2075; splits: train (1517), val (166), test (392); HF EPFL-ECEO/coralscapes; repo https://github.com/eceo-epfl/coralscapesScripts

Metrics

• mean Intersection over Union (mIoU) (primary) — range: [0, 1]
  • For each of the 39 benthic classes, compute IoU = TP / (TP + FP + FN). Average the IoU across all classes to obtain the final score. This is the standard metric for Cityscapes-mimic segmentation benchmarks.

Input / output format

Input: 8-bit PNG images resized to 1024×2048px resolution, representing underwater reef scenes.

Output: Pixel-wise segmentation masks assigning each pixel to one of 39 benthic classes (e.g., live coral, dead coral, substrate, background).

Scoring recipe

def compute_miou(pred_masks, gt_masks, num_classes=39):
    miou_scores = []
    for cls in range(num_classes):
        pred_cls = (pred_masks == cls)
        gt_cls = (gt_masks == cls)
        intersection = np.logical_and(pred_cls, gt_cls).sum()
        union = np.logical_or(pred_cls, gt_cls).sum()
        if union == 0:
            miou_scores.append(1.0)
        else:
            miou_scores.append(intersection / union)
    return np.mean(miou_scores)

Common pitfalls

• Spatial train-test split prevents models from memorizing site-specific features; they must generalize across geographically distinct reefs.
• Conservative annotation protocol means ambiguous or degraded corals are labeled as 'background' or 'dark', requiring models to handle high uncertainty and avoid speculative predictions.
• Severe class imbalance (rarest class has >400x fewer polygons than the most common) can bias models toward majority classes if not properly weighted.

Evidence (verbatim from paper)

We split the dataset spatially by reef site to allow a fair evaluation, resulting in a training set of 1517 images (27 sites), a validation set of 166 images (3 sites), and a test set of 392 images (5 sites).

Citation

@misc{sauder2025coralscapes,
  title={The Coralscapes Dataset: Semantic Scene Understanding in Coral Reefs},
  author={Sauder et al. (2025)},
  year={2025},
  note={arXiv:2503.20000}
}

• arXiv: 2503.20000