Saih Cosmo Scalability Eval

saih-cosmo-scalability-eval

SAIH: A Scalable Evaluation Methodology for Understanding AI Performance Trend on HPC Systems — Jiangsu Du et al. (2022) (arXiv:2212.03410, 2022)

What this evaluates

Evaluates the scalability and performance trends of scientific AI workloads (3D CNNs) on HPC systems under varying node counts and dataset sizes. It probes how hardware constraints like GPU memory, I/O bandwidth, and network communication affect training efficiency and model convergence.

Datasets

• SAIH-cosmo — total 101088; splits: full (101088)

Metrics

• average_flops (primary) — range: Tflops
  • Total floating-point operations computed per GPU divided by the wall-clock execution time for that GPU.
• arithmetic_intensity — range: other
  • Ratio of total FLOPs to total memory accesses (Read + Write). Calculated per kernel or aggregate to measure compute-to-memory ratio.
• training_loss — range: [0, inf)
  • Scalar loss value computed per epoch during training to assess model convergence and accuracy.

Input / output format

Input: Cosmological dataset samples (101,088 total) partitioned into fractions (1/64 to 1/1). 3D CNN models (Small, Medium, Large) with varying parameter counts. Configurations specify node count (1–128), batch size, and precision (FP32/FP16).

Output: Training loss per epoch, per-GPU FLOPs, aggregate FLOPs, memory read/write counts, execution time per epoch, and data loading time percentage.

Scoring recipe

def compute_metrics(flops_per_gpu, exec_time_s, mem_read, mem_write):
    avg_flops = flops_per_gpu / exec_time_s
    arithmetic_intensity = flops_per_gpu / (mem_read + mem_write)
    return avg_flops, arithmetic_intensity

Common pitfalls

• Assuming larger models always yield higher TFLOPs; performance is heavily dictated by arithmetic intensity and memory footprint, not just model size.
• Ignoring I/O bottlenecks during data scaling; PyTorch's data loader can cause severe swapping overhead when datasets exceed node memory capacity.
• Confusing strong scaling (fixed data, varying nodes) with data scaling (fixed nodes, varying data); they probe different bottlenecks (communication vs. I/O).

Evidence (verbatim from paper)

The average flops of a single GPU is shown in Fig. 8(a). ... arithmetic intensity of the small model is much lower than those of the other two models. Here 'Read and 'Write' represent the number of memory read accesses and memory write accesses.

Citation

@misc{du2022saih,
  title={SAIH: A Scalable Evaluation Methodology for Understanding AI Performance Trend on HPC Systems},
  author={Jiangsu Du et al. (2022)},
  year={2022},
  note={arXiv:2212.03410}
}

• arXiv: 2212.03410