qhjqhj00 llm-npu-eval
Fast On-device LLM Inference with NPUs — Xu et al. (2024) (arXiv:2407.05858, 2024)
What this evaluates
This evaluation protocol assesses the performance of an on-device LLM inference framework (llm.npu) on mobile NPUs. It probes the system's ability to accelerate prefill and decoding stages, manage quantization without accuracy loss, and optimize energy efficiency across various LLM sizes and real-world task datasets.
Datasets
- LAMBADA — total ?; splits: test (-1)
- HellaSwag — total ?; splits: test (-1)
- WinoGrande — total ?; splits: test (-1)
- MMLU — total ?; splits: test (-1)
- LongBench — total ?; splits: test (-1)
- DroidTask — total ?; splits: test (-1)
- Persona-Chat — total ?; splits: test (-1)
Metrics
end-to-end latency(primary) — range: seconds- Total time from prompt input to final token generation completion, measured in seconds.
prefill latency— range: seconds- Time taken to process the input prompt and compute KV cache, measured in seconds.
prefill energy consumption— range: joules- Energy used during the prefill stage, measured via Android OS power supply interface profiling every 100ms.
accuracy— range: percent- Percentage of correctly answered questions across multiple-choice and open-ended benchmarks.
Input / output format
Input: Text prompts of varying lengths (64, 256, 1024 tokens for controlled tests; dataset-specific lengths for real-world tasks) fed to the LLM.
Output: Generated text sequences with specified output lengths (e.g., 2–4, 5–11, 1–5, 3–5, or 35–57 tokens depending on the dataset).
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
def compute_latency(start_time, end_time):
return end_time - start_time # seconds
def compute_energy(power_samples, interval=0.1):
return sum(power_samples) * interval # joules
Common pitfalls
- llm.npu uses a CPU backend for decoding, which can bottleneck end-to-end latency compared to baselines that optimize GPU decoding.
- Energy consumption measurements are only available on rootable devices (Redmi K60 Pro), limiting cross-device energy comparisons.
- Shorter prompts (e.g., 64 tokens) cause padding issues and reduce NPU scheduling efficiency, leading to lower speedups than longer prompts.
Evidence (verbatim from paper)
We mainly measure LLM inference accuracy, prefill latency, prefill energy consumption, prefill memory consumption and end-to-end inference latency. The energy consumption is obtained through /sys/class/power_supply in Android OS by profiling every 100ms. All experiments are repeated three times and we report the average numbers.
Citation
@misc{xu2024fastondevice,
title={Fast On-device LLM Inference with NPUs},
author={Xu et al. (2024)},
year={2024},
note={arXiv:2407.05858}
}
- arXiv: 2407.05858