Mlperf Training Eval

mlperf-training-eval

MLPerf Training Benchmark — Mattson et al. (2019) (arXiv:1910.01500, 2019)

What this evaluates

Evaluates the training performance and scalability of machine learning implementations across diverse hardware and software stacks by measuring time to solution under standardized model and hyperparameter configurations.

Datasets

• MLPerf Training Suite — total ?; splits: train (-1); repo https://github.com/mlperf/training

Metrics

• time to solution (primary) — range: other
  • The wall-clock time required to complete a training run or reach a predefined quality target (e.g., accuracy or loss threshold).

Input / output format

Input: Standardized model architectures, hyperparameters, and training procedures provided by the MLPerf reference implementation.

Output: Submission logs containing training duration, convergence curves, and hardware/software configuration details.

Scoring recipe

def score(predictions, gold):
    time = predictions['training_time']
    final_acc = predictions['final_accuracy']
    if final_acc >= gold['target_accuracy']:
        return time
    else:
        return float('inf')

Common pitfalls

• High variance in training time due to inherent randomness in deep learning requires statistical timing rules to mitigate stochasticity.
• Performance optimizations must not degrade model quality or violate standardized hyperparameter constraints.
• Comparisons across submission rounds must account for rule changes (e.g., allowed optimizers) and hardware consistency.

Evidence (verbatim from paper)

MLPerf, like all benchmarks, aims to encourage innovation through constructive competition; we measure progress by comparing results across submission rounds. Figure 4 reveals that the number of chips necessary to produce the fastest time to solution for MLPerf versions v0.5 to v0.6. This number increased by as much as 5.5x .

Citation

@misc{mattson2019mlperf,
  title={MLPerf Training Benchmark},
  author={Mattson et al. (2019)},
  year={2019},
  note={arXiv:1910.01500}
}

• arXiv: 1910.01500