Model Selection Energy Eval

model-selection-energy-eval

Small is Sufficient: Reducing the World AI Energy Consumption Through Model Selection — da Silva Barros et al. (2025) (arXiv:2510.01889, 2025)

What this evaluates

This evaluation protocol assesses the trade-off between model size (parameter count) and task utility across multiple AI benchmarks. It aims to identify energy-efficient models that maintain high performance, enabling estimation of global AI inference energy savings through strategic model selection.

Datasets

• OpenLLM Leaderboard — total ?; splits: (unstated)
• LMSys Chatbot Arena — total ?; splits: (unstated)
• NPHardEval — total ?; splits: (unstated)
• BigCode Leaderboard — total ?; splits: (unstated)
• mtebLeaderboard — total ?; splits: (unstated)
• WMT English-German — total ?; splits: (unstated)
• Open Object Detection Leaderboard — total ?; splits: (unstated)
• ImageNet — total ?; splits: (unstated)
• Semantic Segmentation on ADE20K — total ?; splits: (unstated)
• Open ASR Leaderboard — total ?; splits: (unstated)
• ARCH — total ?; splits: (unstated)
• GenAI — total ?; splits: (unstated)
• MMMU Benchmark — total ?; splits: (unstated)
• Eth1-336 — total ?; splits: (unstated)

Metrics

• Utility (primary) — range: [0, 100] or benchmark-specific
  • Normalized performance score reported by the respective benchmark or leaderboard, used to quantify model capability for a given task.
• Energy(J) — range: other
  • Estimated inference energy consumption in Joules per model-task instance, derived from empirical latency, throughput, or hardware power draw metrics.

Input / output format

Input: Model parameter count, benchmark performance scores, and hardware/inference specifications for each model-task pair.

Output: Utility value, parameter count, and estimated inference energy (Joules) per model, along with selection of 'efficient' vs 'best-performing' models.

Scoring recipe

# 1. Aggregate model performance (Utility) and parameter counts from public leaderboards
# 2. Estimate inference energy (Energy(J)) using empirical latency/throughput or hardware power draw
# 3. Plot Utility vs. parameters to identify Pareto frontier
# 4. Select 'efficient' model: highest Utility-to-size ratio on frontier
# 5. Select 'best' model: absolute maximum Utility
# 6. Compute savings: (Energy_best - Energy_efficient) / Energy_best

Common pitfalls

• Utility scores are benchmark-specific and not directly comparable across different AI tasks.
• Energy estimates rely on hardware assumptions and batch sizes that vary across leaderboards.
• Model adoption is proxied by Hugging Face download counts, which may not reflect actual inference volume.

Evidence (verbatim from paper)

| Task | Model | Params | Utility | Energy(J) | Downloads | | --- | --- | --- | --- | --- | --- | | Text Generation | internlm/internlm2_5-7b-chat (efficient) | 8B | 0.6 | 8035.0 | 37281 |

Citation

@misc{dasilvabarros2025small,
  title={Small is Sufficient: Reducing the World AI Energy Consumption Through Model Selection},
  author={da Silva Barros et al. (2025)},
  year={2025},
  note={arXiv:2510.01889}
}

• arXiv: 2510.01889