Ml Energy Eval

ml-energy-eval

The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — Chung et al. (2025) (arXiv:2505.06371, 2025)

What this evaluates

Measures and optimizes the inference energy consumption of generative AI models under realistic, production-like serving conditions. It evaluates how different hardware and serving configurations affect the trade-off between latency and energy usage, providing automated recommendations for energy-optimal setups.

Datasets

• ML.ENERGY default request dataset — total ?; splits: (unstated); repo https://github.com/ml-energy/benchmark

Metrics

• Energy (Joules/request) (primary) — range: other
  • For diffusion models: Energy_request = Energy_batch / B. For LLMs: Energy_request = (Energy_steady / Tokens_steady) * (1/N) * Σ Tokens_request,i, where steady state is defined as the period when batch size is saturated at the server’s maximum configured batch size.
• Latency (seconds) — range: other
  • Time to generate the full response for diffusion/coding models, or average time per output token for streaming LLM chat models.

Input / output format

Input: Generative model, request dataset (prompts/text/images), and a set of serving configurations to sweep (e.g., GPU model, parallelism, max batch size, denoising steps).

Output: Per-request energy consumption (Joules), per-request latency (seconds), time–energy Pareto frontier across configurations, and the energy-optimal configuration satisfying a user-specified latency constraint.

Scoring recipe

def compute_energy(model_type, requests, config):
    if model_type == 'diffusion':
        energy_batch = measure_energy(config)
        return energy_batch / config.batch_size
    elif model_type == 'llm':
        steady_energy = measure_energy(duration=steady_state_period)
        steady_tokens = count_tokens(duration=steady_state_period)
        energy_per_token = steady_energy / steady_tokens
        return [energy_per_token * req.output_tokens for req in requests]

Common pitfalls

• Ignoring the 'steady state' requirement: energy must be measured only when the server batch size is saturated, not during ramp-up/ramp-down phases.
• Misinterpreting latency constraints: streaming LLMs require time-per-token constraints, while diffusion/coding models require whole-response time constraints.
• Assuming energy scales linearly with batch size without accounting for hardware saturation or power limits.

Evidence (verbatim from paper)

Because the beginning and end of each request are often not aligned with each other, finding each request’s individual energy consumption is non-trivial. For this, we first submit all requests in the request dataset, and as the system runs, identify the steady state as the time period where the batch size is saturated at the server’s maximum configured batch size. This steady state is designed to closely approximate the state of a serving system when it is well-utilized during long-term deployment. Particularly, when the system is ramping up initially with a full queue or ramping down at the end with an empty queue, the server runs with a smaller batch size and does not exhibit the same energy amortization benefits as the steady state. With this, we can derive the average per-request energy consumption with: Energy_request = (Energy_steady / Tokens_steady) * (1/N) * sum(Tokens_request,i).

Citation

@misc{chung2025mlenergy,
  title={The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization},
  author={Chung et al. (2025)},
  year={2025},
  note={arXiv:2505.06371}
}

• arXiv: 2505.06371