Mlperf Mobile Eval

mlperf-mobile-eval

MLPerf Mobile Inference Benchmark — Vijay Janapa Reddi et al. (arXiv:2012.02328, 2020)

What this evaluates

Evaluates mobile AI inference performance across computer vision and NLP tasks on resource-constrained hardware. It measures both model accuracy against strict thresholds and hardware/software stack efficiency under realistic deployment conditions.

Datasets

• ImageNet 2012 validation — total ?; splits: val (-1)
• COCO 2017 validation — total ?; splits: val (-1)
• ADE20K validation — total ?; splits: val (-1)
• SQuAD v1.1 Dev — total ?; splits: dev (-1)

Metrics

• Top-1 accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted classes out of total samples.
• mAP — range: [0, 1]
  • Mean Average Precision across object detection classes, computed over varying IoU thresholds.
• mIoU — range: [0, 1]
  • Mean Intersection over Union across 31 foreground classes plus background, computed per pixel.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for extracted answer spans in question answering.

Input / output format

Input: RGB images resized and normalized to task-specific dimensions (224x224, 300x300, 320x320, or 512x512), or a question paired with a context passage (max 384 tokens).

Output: Class labels, bounding box coordinates with class scores, pixel-wise semantic masks, or extracted text spans.

Scoring recipe

def score(predictions, gold, task):
    if task == 'classification':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task == 'detection':
        return compute_map(predictions, gold, iou_thresh=0.5)
    elif task == 'segmentation':
        ious = [intersection_over_union(p, g) for p, g in zip(predictions, gold)]
        return sum(ious) / len(ious)
    elif task == 'qa':
        prec = sum(1 for p, g in zip(predictions, gold) if g in p) / len(predictions)
        rec = sum(1 for p, g in zip(predictions, gold) if g in p) / len(gold)
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Accuracy thresholds must be met (e.g., 98% of FP32 accuracy) before performance metrics are considered valid.
• Preprocessing steps (resize, crop, normalize) must exactly match the reference implementation to ensure hardware equivalence.
• Vendors must correctly invoke hardware-specific backends (e.g., NNAPI, SNPE) and handle quantization (FP32 vs 8-bit) without altering model stages.

Evidence (verbatim from paper)

We used the COCO 2017 validation data set Lin et al. (2015) and, for the quality metric, the mean average precision (mAP). The target accuracy is an mAP value of 22.7 (93% of FP32 accuracy).

Citation

@misc{reddi2020mlperfmobile,
  title={MLPerf Mobile Inference Benchmark},
  author={Vijay Janapa Reddi et al.},
  year={2020},
  note={arXiv:2012.02328}
}

• arXiv: 2012.02328