Mlperf Power Eval

mlperf-power-eval

MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI — Tschand et al. (2024) (arXiv:2410.12032, 2024)

What this evaluates

Evaluates the energy efficiency of machine learning systems across diverse hardware scales (data center, edge, tiny) and workloads (inference and training). It measures how effectively systems convert electrical energy into computational progress, tracking improvements in samples processed per joule over time and across system configurations.

Datasets

• MLPerf — total 1841; splits: datacenter (590), edge (792), tiny (447), training (12); repo https://github.com/mlcommons/policies

Metrics

• samples per joule (primary) — range: other
  • Total number of inference samples (or tokens converted to samples) processed divided by the total energy consumed in joules. Often reported inversely as Joules per Sample (J/Sample) for direct energy cost comparison.
• time to solution — range: other
  • Wall-clock time required to complete the training or inference workload.

Input / output format

Input: System configuration (hardware, accelerators, interconnects), workload definition (model architecture, dataset, batch size, precision), and power/energy logs from verified MLPerf submission runs.

Output: Normalized energy efficiency (samples per joule), absolute energy consumption (J/Sample), and time-to-solution metrics.

Scoring recipe

def compute_energy_efficiency(total_samples, total_energy_joules):
    if total_energy_joules <= 0:
        return 0.0
    return total_samples / total_energy_joules
# For LLMs, convert tokens to samples using median sequence length
# samples = tokens / median_sequence_length

Common pitfalls

• Comparing energy efficiency across different hardware scales (datacenter vs edge vs tiny) without accounting for workload complexity and input size differences.
• Using inconsistent performance metrics for LLMs (samples/second vs tokens/second) when calculating energy efficiency, leading to non-comparable J/Sample values.
• Ignoring interconnect and networking energy costs when scaling training systems, which significantly impacts total energy consumption despite reduced wall-clock time.

Evidence (verbatim from paper)

These advancements are expressed using normalized samples per joule, a measure that encapsulates enhancements in processing power alongside an industry-wide emphasis on optimizing AI technology for energy efficiency.

Citation

@misc{tschand2024mlperfpower,
  title={MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI},
  author={Tschand et al. (2024)},
  year={2024},
  note={arXiv:2410.12032}
}

• arXiv: 2410.12032