On Device LLM Energy Eval

on-device-llm-energy-eval

Sustainability Is Not Linear: Quantifying Performance, Energy, and Privacy Trade-offs in On-Device Intelligence — Eziyo Ehsani et al. (arXiv:2603.26603, 2026)

What this evaluates

Evaluates the trade-offs between inference speed, energy consumption, latency, and generation quality of various LLM architectures and quantization schemes running on mobile hardware. It specifically measures how model size, sparsity (MoE), and compression formats impact physical battery drain and user-perceived responsiveness.

Datasets

• Summarization Task (On-Device Profiling) — total ?; splits: (unstated)

Metrics

• Energy per Token (Joules) (primary) — range: other
  • Physical energy consumed in Joules divided by the number of generated tokens, measured via hardware telemetry.
• G-Eval — range: other
  • Automated quality assessment of generated summaries using the G-Eval framework, measuring semantic coherence and abstraction.
• End-to-End Latency — range: other
  • Total time from prompt input to generation of up to 100 output tokens, decomposed into prefetch and generation phases.

Input / output format

Input: Text prompts for summarization tasks.

Output: Autoregressive text generation up to 100 tokens.

Scoring recipe

for model in models:
    for quant in [Q4_K_M, IQ4_XS]:
        energies, latencies, throughputs = [], [], []
        for run in range(30):
            start = time()
            e_start = read_hardware_telemetry()
            output = model.generate(prompt, max_tokens=100)
            e_end = read_hardware_telemetry()
            latencies.append(time() - start)
            energies.append(e_end - e_start)
            throughputs.append(100 / (time() - start))
        # Compute medians; apply Shapiro-Wilk, Friedman, Wilcoxon signed-rank with Holm-Bonferroni correction

Common pitfalls

• Assuming 4-bit quantization (IQ4_XS) reduces energy consumption on CPU-only mobile inference; unpacking overhead offsets memory savings.
• Ignoring the memory-bound sequential nature of the generation phase, which dominates latency and energy over the parallel prefill phase.
• Assuming model size linearly scales energy consumption; MoE architectures decouple total parameters from active compute and energy draw.

Evidence (verbatim from paper)

Higher generation speed almost universally corresponds to lower cognitive quality as measured by the G-Eval framework.

Citation

@misc{ehsani2026sustainability,
  title={Sustainability Is Not Linear: Quantifying Performance, Energy, and Privacy Trade-offs in On-Device Intelligence},
  author={Eziyo Ehsani et al.},
  year={2026},
  note={arXiv:2603.26603}
}

• arXiv: 2603.26603