Mlperf Hpc Eval

mlperf-hpc-eval

MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems — Farrell et al. (2021) (arXiv:2110.11466, 2021)

What this evaluates

Evaluates end-to-end performance of HPC systems for scientific machine learning, focusing on data staging, I/O efficiency, and model convergence under massive dataset constraints. It measures how well systems handle large-scale volumetric and high-resolution image workloads while meeting strict accuracy targets.

Datasets

• CosmoFlow — total ?; splits: train (262144), test (65536); repo https://github.com/mlcommons/hpc
• DeepCAM — total ?; splits: train (121266), test (15158); repo https://github.com/mlcommons/hpc

Metrics

• MAE (primary) — range: [0, inf)
  • Mean absolute error between predicted and true cosmological parameters across the test set.
• IOU (primary) — range: [0, 1]
  • Intersection-over-Union between predicted segmentation masks and ground truth masks, averaged across classes.

Input / output format

Input: CosmoFlow: 3D volumetric cubes of size 128^3 with 4 channels. DeepCAM: 16-channel tensors of size 1152x768.

Output: CosmoFlow: 4 predicted cosmological parameters. DeepCAM: 1152x768 segmentation masks for 3 classes.

Scoring recipe

def score_cosmoflow(y_pred, y_true):
    mae = np.mean(np.abs(y_pred - y_true))
    assert mae < 0.124, 'Quality target not met'
    return mae

def score_deepcam(y_pred, y_true):
    intersection = np.sum((y_pred == y_true) & (y_pred > 0), axis=(1,2,3))
    union = np.sum((y_pred > 0) | (y_true > 0), axis=(1,2,3))
    iou = np.mean(intersection / union)
    assert iou > 0.82, 'Quality target not met'
    return iou

Common pitfalls

• I/O bottlenecks often dominate end-to-end runtime, making storage and interconnect performance more critical than raw compute throughput.
• Runs are only valid if the strict quality targets (MAE < 0.124 or IOU > 0.82) are met; otherwise, timing measurements are discarded.
• CosmoFlow requires splitting 512^3 volumes into 128^3 cubes due to memory constraints, which affects data loading and parallelism strategies.

Evidence (verbatim from paper)

The target quality is chosen to be mean-absolute-error (MAE) < 0.124 when scaling the batch size and learning rate above the reference configuration. CosmoFlow training exhibited high variability in the number of epochs to converge, which motivated a requirement of 10 training runs to get a reliable measurement of the time to train. ... The target score is the intersection-over-union (IOU) between the predictions and the targets. The scientifically motivated target score is 0.82, which corresponds to a similarity of 82%.

Citation

@misc{farrell2021mlperf,
  title={MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems},
  author={Farrell et al. (2021)},
  year={2021},
  note={arXiv:2110.11466}
}

• arXiv: 2110.11466