Climate Segmentation Eval

climate-segmentation-eval

Exascale Deep Learning for Climate Analytics — Kurth et al. (2018) (arXiv:1810.01993, 2018)

What this evaluates

This evaluation probes pixel-level weather pattern segmentation (atmospheric rivers and tropical cyclones) from multi-channel climate data. It measures both segmentation accuracy and exascale training throughput/scaling efficiency across different network architectures and hardware configurations.

Datasets

• Climate weather pattern dataset — total ?; splits: validation (-1)

Metrics

• IoU (primary) — range: [0, 1]
  • Intersection over Union: the area of overlap between the predicted segmentation mask and the ground truth mask divided by the area of union between them.
• Sustained Throughput — range: PF/s
  • Training performance measured in petaflops per second (PF/s) or samples per second, calculated from floating-point operations per image and sustained training rate per GPU.
• Parallel Efficiency — range: percent
  • Ratio of actual scaled throughput to ideal linear scaling throughput, expressed as a percentage.

Input / output format

Input: 16-channel climate data images (e.g., integrated water vapor) representing weather patterns.

Output: Pixel-level segmentation masks classifying atmospheric features (e.g., atmospheric rivers in blue, tropical cyclones in red).

Scoring recipe

def compute_iou(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    if union == 0:
        return 1.0
    return intersection / union

Common pitfalls

• The weighted loss function penalizes false negatives on tropical cyclones ~37x more than false positives, which can inflate IoU by overpredicting features.
• FP16 training shows lower compute efficiency than FP32 due to memory-bound kernels, despite higher theoretical throughput.
• Scaling experiments without local data staging suffer significant efficiency drops (>9.5%) at large node counts due to filesystem bandwidth limits.

Evidence (verbatim from paper)

Segmentation accuracy is often measured using the intersection over union (IoU) metric. The Tiramisu network obtained an IoU of 59% on our validation data set, while our modified DeepLabv3+ network was able to achieve 73% IoU.

Citation

@misc{kurth2018exascale,
  title={Exascale Deep Learning for Climate Analytics},
  author={Kurth et al. (2018)},
  year={2018},
  note={arXiv:1810.01993}
}

• arXiv: 1810.01993