qhjqhj00 r-ice-eval
Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs — Sikand et al. (2025) (arXiv:2506.08727, 2025)
What this evaluates
Evaluates the accuracy of a regression framework (R-ICE) in estimating prompt-level inference carbon and energy emissions for LLMs using only token counts and publicly available performance data. It probes whether runtime can be reliably modeled as a piecewise linear function of input and output tokens without intrusive monitoring or architecture details.
Datasets
- HELM — total ?; splits: test (-1)
Metrics
average prediction error(primary) — range: percent- Mean absolute percentage error between the estimated and actual inference carbon/energy emissions across all benchmark instances. Calculated as the average of |estimated - actual| / actual for each prompt.
Input / output format
Input: Prompt text and generated output text (or token counts) from HELM benchmark instances.
Output: Estimated inference runtime (Idealized) and corresponding carbon/energy emissions per instance.
Scoring recipe
errors = []
for instance in dataset:
actual = get_actual_emission(instance)
estimated = estimate_emission(instance.prompt_tokens, instance.output_tokens)
errors.append(abs(estimated - actual) / actual)
return mean(errors) * 100
Common pitfalls
- Assumes Idealized runtime on standardized hardware, ignoring real-world variations in GPU utilization, batch sizes, and system overheads.
- Relies solely on token counts for runtime estimation, which may not capture latency spikes from KV-cache management or network I/O in distributed inference.
Evidence (verbatim from paper)
R-ICE introduces a benchmark-based regression framework to estimate prompt-level inference carbon and energy emissions in LLMs, leveraging publicly available performance data from HELM benchmarks. It avoids reliance on model architecture details or intrusive monitoring by modeling runtime as a function of prompt and output token counts, achieving ~15% average prediction error while enabling non-intrusive, scalable estimation for enterprise-level carbon accounting and dynamic routing decisions.
Citation
@misc{sikand2025breaking,
title={Breaking the ICE: Exploring promises and challenges of benchmarks for Inference Carbon & Energy estimation for LLMs},
author={Sikand et al. (2025)},
year={2025},
note={arXiv:2506.08727}
}
- arXiv: 2506.08727