Mobile Dl Inference Eval

mobile-dl-inference-eval

Deep Learning Inference on Heterogeneous Mobile Processors: Potentials and Pitfalls — Sicong Liu et al. (arXiv:2405.01851, 2024)

What this evaluates

Evaluates the effectiveness of parallel deep learning inference strategies across heterogeneous mobile processors (CPU, GPU, DSP) under varying workloads and dynamic system conditions. It probes how operator support, scheduling granularity, and competing processes impact inference latency, resource utilization, and system responsiveness.

Datasets

• Standard DL Models (YOLOv2, VGG-16, PoseNet, FST, RetinaFace, ResNet-18, ResNet-50) — total ?; splits: test (-1)

Metrics

• inference latency (ms) (primary) — range: ms
  • Direct empirical measurement of wall-clock time to complete a single inference pass or frame processing on the target device.
• speedup (%) — range: percent
  • Calculated relative to a baseline strategy: (baseline_latency - method_latency) / baseline_latency * 100.
• top accuracy (%) — range: percent
  • Standard top-1 classification accuracy on the model's validation/test set.
• frame drop rate (%) — range: percent
  • Percentage of dropped frames during concurrent UI/video playback while running DL inference.

Input / output format

Input: Deep learning model architectures executed on mobile SoCs (Snapdragon 855/870, Kirin 985) under varying system states (competing processes, temperature, cache contention, data types).

Output: Measured inference latency in milliseconds, speedup percentage, top-1 accuracy percentage, memory usage in MB, and frame drop rate percentage.

Scoring recipe

def calculate_speedup(gold_baseline_latency_ms, pred_method_latency_ms):
    return ((gold_baseline_latency_ms - pred_method_latency_ms) / gold_baseline_latency_ms) * 100

def calculate_frame_drop_rate(dropped_frames, total_frames):
    return (dropped_frames / total_frames) * 100

# Accuracy is computed as standard top-1 classification ratio
accuracy = correct_predictions / total_predictions

Common pitfalls

• Assuming cross-processor parallel inference always yields lower latency; scheduling overhead and data transfer can make single-processor execution faster.
• Ignoring dynamic resource conditions (temperature throttling, competing processes, cache contention) which cause significant latency variance compared to offline benchmarks.
• Overlooking operator fallbacks to CPU when accelerators lack support, which negates hardware acceleration benefits and increases latency.

Evidence (verbatim from paper)

We compare the inference latency of six strategies: ① Mace framework executes inference on CPU, ② Mace on GPU with the buffer type, ③ Mace on GPU with the image type, ④ μ Layer on CPU+GPU, ⑤ CoDL on CPU+GPU in parallel with buffer type, and ⑥ CoDL on CPU & GPU in parallel with the image type. We test them on five models, i.e., YOLOv2, VGG-16, PoseNet, Fast Style Transfer (FST), and RetinaFace, across three mobile devices, i.e., Snakeragon 855 (D1), Snakeragon 870 (D7), and Kirin 985 (D2).

Citation

@misc{liu2024deeplearninginference,
  title={Deep Learning Inference on Heterogeneous Mobile Processors: Potentials and Pitfalls},
  author={Sicong Liu et al.},
  year={2024},
  note={arXiv:2405.01851}
}

• arXiv: 2405.01851