Edge Lm Inference Eval

edge-lm-inference-eval

Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge — Abstreiter et al. (2025) (arXiv:2503.09114, 2025)

What this evaluates

Evaluates the feasibility and performance trade-offs of running small generative language models on edge hardware. It probes memory constraints, inference latency, token throughput, and energy efficiency across different quantization schemes and system configurations.

Datasets

• None (system-level inference benchmark) — total ?; splits: (unstated)

Metrics

• generation_throughput (primary) — range: other
  • Number of tokens generated divided by the time spent in the generation phase.
• end_to_end_latency — range: other
  • Total time from input prompt submission to the completion of the generation phase.
• energy_per_token — range: other
  • Total energy consumed during inference divided by the total number of tokens generated.
• peak_memory_usage — range: other
  • Maximum RAM or VRAM consumed during model loading and inference execution.

Input / output format

Input: Text prompt (context) provided to the language model for autoregressive generation.

Output: Autoregressively generated text tokens until a stop condition or maximum length is reached.

Scoring recipe

def calc_generation_throughput(tokens_generated, gen_time_sec):
    return tokens_generated / gen_time_sec

def calc_energy_per_token(total_energy_joules, tokens_generated):
    return total_energy_joules / tokens_generated

def calc_ttft(load_time_sec, prefetch_time_sec):
    return load_time_sec + prefetch_time_sec

Common pitfalls

• Ignoring the warm-up run: The first run is excluded from reported results, but first-run load times can be 5–200 seconds longer than cached runs.
• Assuming maximum thread counts optimize performance: Using all available threads often degrades generation throughput due to cache misses and excessive context switches.
• Overlooking memory exhaustion: Larger models or extended context sizes cause page faults or device crashes, drastically altering latency and energy metrics.

Evidence (verbatim from paper)

For CPU inference, prefetch phase throughput scales roughly linearly with increasing thread count, with performance and on-demand governors yielding higher throughputs than the powersave governor. Similarly, generation throughput also increases linearly with thread count but eventually plateaus.

Citation

@misc{abstreiter2025edgeinference,
  title={Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge},
  author={Abstreiter et al. (2025)},
  year={2025},
  note={arXiv:2503.09114}
}

• arXiv: 2503.09114