Mobile Vlm Deployment Eval

mobile-vlm-deployment-eval

Efficient Deployment of Vision-Language Models on Mobile Devices: A Case Study on OnePlus 13R — Guerrero et al. (2025) (arXiv:2507.08505, 2025)

What this evaluates

Evaluates the runtime efficiency, hardware utilization, and thermal/energy impact of deploying vision-language models on mobile devices. It measures latency breakdowns, CPU/GPU/NPU usage, power consumption, and output characteristics across different inference frameworks.

Datasets

• Custom Mobile VLM Inference Test Set — total ?; splits: (unstated)

Metrics

• Latency (primary) — range: other
  • Wall-clock time for the entire inference pipeline, measured from model load to the end of token generation.
• Stage Latency — range: other
  • Time spent in each pipeline stage: model loading, image encoding, prompt evaluation, and autoregressive token generation.
• Hardware Utilization — range: percent
  • Percentage of time the CPU, GPU, or NPU is actively processing tasks during inference, reported as aggregate across all cores.
• Power Consumption — range: other
  • Average or peak power draw during the inference run, derived from battery current measurements.
• Die Temperature — range: other
  • Peak or steady-state chip temperature recorded during inference.

Input / output format

Input: An image and a text prompt fed to a VLM running on a mobile device (OnePlus 13R) via a specific inference framework (llama.cpp, mllm, or MLC-Imp).

Output: Generated text response, plus system profiling logs capturing latency per stage, CPU/GPU/NPU utilization percentages, power draw in watts, and die temperature in Celsius.

Scoring recipe

def compute_metrics(profiling_log):
    latency_total = profiling_log['total_time_ms']
    latency_stages = {k: v for k, v in profiling_log.items() if k in ['model_load', 'image_encoding', 'prompt_eval', 'token_gen']}
    cpu_util = profiling_log['cpu_percent']
    gpu_util = profiling_log['gpu_percent']
    power_w = profiling_log['power_watts']
    temp_c = profiling_log['temp_celsius']
    output_tokens = len(profiling_log['generated_text'].split())
    return {
        'total_latency_ms': latency_total,
        'stage_latencies_ms': latency_stages,
        'cpu_util_percent': cpu_util,
        'gpu_util_percent': gpu_util,
        'power_w': power_w,
        'temp_c': temp_c,
        'output_tokens': output_tokens
    }

Common pitfalls

• Latency and power measurements are highly sensitive to background processes and thermal throttling on mobile devices; results may vary significantly across runs without controlled environmental conditions.
• Hardware utilization percentages are reported as aggregate across all cores (e.g., 600% for 8 cores), which can be misleading if not normalized per-core or compared against baseline idle states.
• Power and temperature readings are device-specific (OnePlus 13R with Adreno 740 GPU and Hexagon NPU) and may not generalize to other mobile SoCs or thermal envelopes.

Evidence (verbatim from paper)

We analyze latency by decomposing the vision-language pipeline into four stages: model loading, image encoding, prompt evaluation, and autoregressive token generation. Significant differences emerged in power consumption and thermal profiles (Fig. 4). CPU-only LLaVA stacks were notably hotter and more power-intensive (10–12 W, 90–95°C).

Citation

@misc{guerrero2025efficient,
  title={Efficient Deployment of Vision-Language Models on Mobile Devices: A Case Study on OnePlus 13R},
  author={Guerrero et al. (2025)},
  year={2025},
  note={arXiv:2507.08505}
}

• arXiv: 2507.08505