Text Classification Energy Eval

text-classification-energy-eval

Comparing energy consumption and accuracy in text classification inference — Zschache et al. (2025) (arXiv:2508.14170, 2025)

What this evaluates

This benchmark evaluates the trade-off between model accuracy, inference energy consumption, and runtime across diverse text classification models and hardware configurations. It probes whether larger or more complex models consistently outperform smaller or traditional ones in accuracy while highlighting the energy costs of different architectures and deployment strategies.

Datasets

• Text classification test set — total 189; splits: test (189)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the fraction of correctly predicted labels out of the total number of samples in the test set.
• energy_consumption — range: Wh
  • Total electrical energy consumed during the inference step, measured in watt-hours (Wh). Measured directly via hardware monitoring tools while classifying 189 samples.
• inference_duration — range: seconds
  • Wall-clock time required to classify 189 samples, measured in seconds. Strictly excludes model loading and data preprocessing time.

Input / output format

Input: A batch of 189 text samples for classification.

Output: Predicted class label for each of the 189 text samples.

Scoring recipe

def compute_metrics(predictions, gold_labels, energy_wh, duration_s):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = correct / len(gold_labels)
    return accuracy, energy_wh, duration_s

# Protocol: Run inference on 10 different test sets (189 samples each).
# Average accuracy, energy, and duration across the 10 runs.
# Report mean and variance (accuracy variance < 0.002, energy/duration variance < 0.2 dex).

Common pitfalls

• Energy and duration must be measured strictly during the inference step, explicitly excluding model loading and data preprocessing time.
• Increasing the number of GPUs does not guarantee lower energy consumption; it can increase total energy due to hardware overhead and inter-node communication costs.
• Runtime is only a reliable proxy for energy consumption when the hardware configuration (GPU type, count, and node topology) is held constant.

Evidence (verbatim from paper)

For each model, we report accuracy, energy consumption, and inference duration. The energy consumption and duration were measured only for the inference step, i.e., after the model and data were already loaded. One inference run involves classifying 189 text samples from a test set. All tables and figures present the average results over 10 runs on different test sets, with the same 10 test sets used for each model.

Citation

@misc{zschache2025energy,
  title={Comparing energy consumption and accuracy in text classification inference},
  author={Zschache et al. (2025)},
  year={2025},
  note={arXiv:2508.14170}
}

• arXiv: 2508.14170