Melt Eval

melt-eval

MELTing point: Mobile Evaluation of Language Transformers — Laskaridis et al. (2024) (arXiv:2403.12844, 2024)

What this evaluates

Evaluates the on-device runtime performance, energy efficiency, thermal behavior, and quality of experience of LLM inference across mobile and edge platforms under varying quantization and framework configurations.

Datasets

• OpenAssistant/oasst1 (filtered subset) — total 50; splits: test (50); HF OpenAssistant/oasst1

Metrics

• throughput (primary) — range: other
  • Total number of generated tokens divided by the total time spent in the generation phase (tokens/sec). Measures the memory-bound decoding speed of the model on-device.
• energy_discharge_per_token — range: other
  • Total battery discharge measured in mAh divided by the total number of generated tokens. Quantifies the energy cost of decoding a single token.
• model_loading_time — range: other
  • Wall-clock time elapsed from initiating model load to the model being ready for inference (seconds). Impacts device responsiveness and QoE.

Input / output format

Input: Conversational prompts from the filtered OpenAssistant subset, featuring 6–10 turns per conversation and prompt lengths typically under 36 words. Context size and maximum generation length are configurable via grid search.

Output: Variable-length generated text tokens, capped at a maximum length of 64, 128, or 256 tokens depending on the experimental configuration. Micro-experiments fix output length and ignore tokens.

Scoring recipe

# Throughput (tokens/sec)
throughput = total_generated_tokens / generation_time_sec

# Energy per token (mAh/token)
energy_per_token = total_battery_discharge_mAh / total_generated_tokens

# Model loading time (sec)
load_time = time_inference_ready - time_load_started

# QoE degradation (throughput drop over continuous prompts)
degradation = (throughput[i] - throughput[i+1]) / throughput[i]

Common pitfalls

• Confusing prefill throughput with generation throughput; generation is typically memory-bound and significantly slower due to KV-cache usage.
• Ignoring thermal throttling and DVFS effects, which cause non-linear throughput degradation during continuous inference sessions.
• Assuming GPU execution always outperforms CPU on mobile; the paper notes CPU can be comparable or better for certain quantized models on Android due to framework/driver overhead.

Evidence (verbatim from paper)

We divide our results per device tier and illustrate the average throughput (in tokens/sec) per framework in Fig[4]. Next, we take the same set of models and illustrate the energy discharge (in mAh) per token generated across devices and frameworks in Fig.[5].

Citation

@misc{laskaridis2024melt,
  title={MELTing point: Mobile Evaluation of Language Transformers},
  author={Laskaridis et al. (2024)},
  year={2024},
  note={arXiv:2403.12844}
}

• arXiv: 2403.12844