Mlperf Inference Eval

mlperf-inference-eval

MLPerf Inference Benchmark — Vijay Janapa Reddi et al. (arXiv:1911.02549, 2019)

What this evaluates

Evaluates ML inference systems across diverse hardware and software stacks under realistic deployment scenarios. It measures both model quality against strict baselines and system performance (latency/throughput) to enable architecture-neutral comparisons of production-like workloads.

Datasets

• ImageNet — total ?; splits: test (-1)
• COCO — total ?; splits: test (-1)
• WMT16 EN-DE — total ?; splits: test (-1)

Metrics

• Top-1 accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels out of total samples. MLPerf requires achieving 99% of the FP32 baseline accuracy before system metrics are recorded.
• mAP — range: [0, 1]
  • Mean Average Precision for object detection bounding boxes. Must reach 99% of the FP32 baseline (e.g., 0.20) to qualify for latency measurement.
• SacreBleu — range: [0, 100]
  • BLEU score computed with standardized tokenization and reference handling. Must reach 99% of the FP32 baseline (e.g., 23.9) to qualify.

Input / output format

Input: Image tensors (224x224, 300x300, or 1200x1200) or text sequences for machine translation.

Output: Predicted class labels, bounding boxes with confidence scores, or translated text sequences.

Scoring recipe

def evaluate(predictions, gold, baseline_accuracy):
    if metric == 'Top-1 accuracy':
        score = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif metric == 'mAP':
        score = compute_map(predictions, gold)
    elif metric == 'SacreBleu':
        score = sacrebleu.corpus_bleu(predictions, [gold])
    
    # Quality gate: must meet 99% of FP32 baseline
    if score < 0.99 * baseline_accuracy:
        return {'valid': False, 'score': score}
    
    # Measure latency/throughput per deployment scenario
    latency = measure_latency(predictions, gold)
    return {'valid': True, 'accuracy': score, 'latency': latency}

Common pitfalls

• Failing the quality gate: latency/throughput measurements are invalid if the model does not first achieve 99% of the FP32 baseline accuracy/mAP/SacreBleu.
• Ignoring scenario constraints: single-stream, multistream, server, and offline scenarios enforce strict data-availability and batching rules that drastically change system performance.
• Framework-specific operator differences (e.g., NMS variants) can cause accuracy drops during model conversion, requiring careful validation before benchmarking.

Evidence (verbatim from paper)

These models vary tremendously in compute and memory requirements (e.g., a 50× difference in gigaflops), while the corresponding Top-1 accuracy ranges from 55% to 83%.

Citation

@misc{reddi2019mlperf,
  title={MLPerf Inference Benchmark},
  author={Vijay Janapa Reddi et al.},
  year={2019},
  note={arXiv:1911.02549}
}

• arXiv: 1911.02549