qhjqhj00 milabench-eval
Introducing Milabench: Benchmarking Accelerators for AI — Delaunay et al. (2024) (arXiv:2411.11940, 2024)
What this evaluates
Evaluates the real-world computational performance and software stack maturity of AI accelerators across diverse workloads including NLP, computer vision, reinforcement learning, and graph neural networks.
Datasets
- Milabench — total ?; splits: (unstated)
Metrics
performance(primary) — range: other- Throughput or execution speed of a workload on a target GPU divided by the same workload's performance on the NVIDIA A100 baseline. Values >1 indicate faster execution or higher throughput than the baseline.
Input / output format
Input: Standardized end-to-end AI workloads (training, fine-tuning, or inference) implemented in PyTorch/JAX, including model definitions, data loaders, and hyperparameters, executed on the target GPU node.
Output: Execution throughput (e.g., tokens/sec, images/sec, or FLOPS) and relative performance compared to the A100 baseline.
Scoring recipe
def compute_performance(target_perf, baseline_perf):
if baseline_perf == 0:
return float('inf')
return target_perf / baseline_perf
# Aggregate across workloads by averaging relative_performance values
Common pitfalls
- Synthetic FLOP benchmarks (e.g., peak TF32/FP16) often fail to predict real-world workload performance due to software stack maturity and kernel optimization differences.
- Vendor default precision settings (e.g., Gaudi2 defaulting to TF32) can mask true hardware capabilities in standard benchmarks.
- Framework and library updates (e.g., new PyTorch versions) frequently introduce performance gains that vary by vendor, making results time-sensitive.
Evidence (verbatim from paper)
The performance results from the A100 were used as the baseline for comparison with the H100, MI300X, and Gaudi2 GPUs.
Citation
@misc{delaunay2024milabench,
title={Introducing Milabench: Benchmarking Accelerators for AI},
author={Delaunay et al. (2024)},
year={2024},
note={arXiv:2411.11940}
}
- arXiv: 2411.11940