Mdlbench Eval

mdlbench-eval

Benchmarking of DL Libraries and Models on Mobile Devices — Zhang et al. (2022) (arXiv:2202.06512, 2022)

What this evaluates

Evaluates the inference performance and overhead of six major mobile deep learning libraries across diverse model architectures, tasks, and mobile hardware configurations. It measures how software-level optimizations and library choices impact on-device latency compared to hardware capabilities and algorithmic optimizations like quantization.

Datasets

• MDLBench — total 15; splits: test (15)

Metrics

• inference time (primary) — range: other
  • Average latency measured over N iterations (default N=50) after a warm-up phase. Includes per-operator latency and duration traces collected via library instrumentation or adb.

Input / output format

Input: Pre-trained deep learning models (e.g., MobileNet, ResNet, YOLO, ALBERT) converted to library-specific formats, executed on mobile devices with specified hardware configurations (CPU/GPU/DSP) and precision levels (float32/int8).

Output: Inference latency (time per inference), per-operator latency/duration traces, and input/output dimensions, written to device storage and retrieved to desktop.

Scoring recipe

def compute_avg_latency(model, lib, device, iterations=50):
    # 1. Push lib and model to device via adb
    # 2. Kill background apps, set CPU freq, use 4 big cores
    # 3. Load lib and model into memory (warm-up)
    latencies = []
    for _ in range(iterations):
        t_start = current_time()
        lib.run(model)
        t_end = current_time()
        latencies.append(t_end - t_start)
    # 4. Retrieve results from device storage
    return sum(latencies) / len(latencies)

Common pitfalls

• Performance varies significantly across hardware accelerators (CPU/GPU/DSP) and precision (float32 vs int8); results are not directly comparable without specifying the exact configuration.
• The benchmark measures raw inference latency, not model accuracy or task performance, so it does not evaluate the correctness of the models.
• Background processes and system thermal throttling can affect results; the protocol explicitly requires killing background apps and controlling CPU frequency.

Evidence (verbatim from paper)

MDLBench profiles the inference time and operator-level information, e.g., per-operator latency, duration, input/output dimension, etc.

Citation

@misc{zhang2022mdlbench,
  title={Benchmarking of DL Libraries and Models on Mobile Devices},
  author={Zhang et al. (2022)},
  year={2022},
  note={arXiv:2202.06512}
}

• arXiv: 2202.06512