Edge LLM Energy Accuracy Eval

edge-llm-energy-accuracy-eval

Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency — Husom et al. (2025) (arXiv:2504.03360, 2025)

What this evaluates

Evaluates the trade-offs between quantization levels, model architectures, and task characteristics on energy efficiency and output accuracy for LLMs deployed on edge hardware.

Datasets

• bigbenchhard — total ?; splits: test (-1)
• commonsenseqa — total ?; splits: test (-1)
• gsm8k — total ?; splits: test (-1)
• humaneval — total ?; splits: test (-1)
• truthfulqa — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard dataset accuracy calculated as the proportion of correctly predicted instances out of the total number of instances.
• energy_per_token — range: Joules
  • Total hardware energy consumption in Joules divided by the number of tokens generated in the response.

Input / output format

Input: Text prompt with task-specific instructions (e.g., math problems, code generation, multiple-choice questions) fed to the LLM.

Output: Model-generated text response (variable length in tokens).

Scoring recipe

def compute_metrics(predictions, golds, response_tokens, total_joules):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    accuracy = correct / len(golds)
    energy_per_token = total_joules / response_tokens
    return {'accuracy': accuracy, 'energy_per_token': energy_per_token}

Common pitfalls

• Aggressive quantization (e.g., q3) does not always yield lower energy due to computational overhead; q4 variants can be more efficient.
• Response length heavily skews per-token energy metrics; longer outputs distribute fixed inference overhead, making direct per-token comparisons misleading.
• Highly quantized models exhibit high variance in energy consumption, requiring multiple runs or careful averaging for reliable profiling.

Evidence (verbatim from paper)

To address RQ2, we analyzed the accuracy-energy tradeoff across quantization levels and model families. Table[8] reports accuracy scores for all models on the five benchmark datasets, while Figure[5] visualizes accuracy comparisons across variants.

Citation

@misc{husom2025sustainable,
  title={Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency},
  author={Husom et al. (2025)},
  year={2025},
  note={arXiv:2504.03360}
}

• arXiv: 2504.03360