Mlperf Automotive Eval

mlperf-automotive-eval

MLPerf Automotive — Shojaei et al. (2025) (arXiv:2510.27065, 2025)

What this evaluates

Evaluates real-time perception capabilities for automotive systems, specifically 2D object detection, 2D semantic segmentation, and 3D object detection. It measures how well inference engines meet strict latency constraints and accuracy tolerances required for safety-critical driving tasks.

Datasets

• nuScenes — total ?; splits: full (-1)
• MLCommons Cognata Dataset — total ?; splits: full (-1)

Metrics

• tail_latency (primary) — range: other
  • 99.9th percentile of end-to-end inference latency per query across the entire dataset run. Measured in milliseconds.
• accuracy_vs_fp32 — range: percent
  • Percentage of the FP32 reference model's accuracy achieved by the submitted model, calculated as (submitted_accuracy / fp32_reference_accuracy) * 100.

Input / output format

Input: Multi-camera image sequences: 6 images at 800×450 resolution for BEVFormer; single high-resolution image at 3840×2160 for SSD and DeepLabv3+.

Output: Per-query inference results: bounding boxes and class probabilities for detection tasks, or pixel-wise semantic labels for segmentation tasks.

Scoring recipe

def compute_metrics(predictions, gold, fp32_predictions):
    # 1. Compute tail latency (99.9th percentile)
    latencies = [get_inference_time(q) for q in dataset]
    tail_latency = np.percentile(latencies, 99.9)
    
    # 2. Compute accuracy relative to FP32 reference
    submitted_acc = evaluate_predictions(predictions, gold)
    fp32_acc = evaluate_predictions(fp32_predictions, gold)
    accuracy_pct = (submitted_acc / fp32_acc) * 100
    
    return tail_latency, accuracy_pct

Common pitfalls

• Closed-division submissions strictly prohibit retraining, caching results, and benchmark-aware preprocessing.
• Accuracy is evaluated relative to the FP32 reference model's score, not against absolute ground-truth thresholds.
• Latency is measured at the 99.9th percentile (tail latency), not average or median, requiring longer runtimes for statistical validity.
• Synthetic dataset (Cognata) may have unrepresentative object densities affecting NMS-heavy models compared to real-world data.

Evidence (verbatim from paper)

We use tail latency as our performance metric, in a similar way to the latency-sensitive inference benchmarks used in the MLPerf Inference Edge category. However, we need to set stronger requirements. There is a practical trade off in that the stronger the requirements, the longer the benchmark must be run for a valid submission because estimates of the tail latency require more samples with each decimal of precision we require. We settled on a tail latency of 99.9% performance metric.

Citation

@misc{shojaei2025mlperfautomotive,
  title={MLPerf Automotive},
  author={Shojaei et al. (2025)},
  year={2025},
  note={arXiv:2510.27065}
}

• arXiv: 2510.27065