Intelligence Per Watt Eval

intelligence-per-watt-eval

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon et al. (arXiv:2511.07885, 2025)

What this evaluates

Evaluates the efficiency of local LLM inference by combining task accuracy with energy consumption to compute Intelligence per Watt (IPW). It probes how model architecture, hardware acceleration, and numerical precision affect the trade-off between performance and power usage on real-world chat and reasoning tasks.

Datasets

• WildChat — total 1000000; splits: test (1000000)
• NaturalReasoning — total 10000; splits: test (10000)
• SuperGPQA — total 10000; splits: test (10000)
• MMLU Pro — total 10000; splits: test (10000)

Metrics

• accuracy (primary) — range: percent
  • Percentage of queries where the model's response is judged correct according to the benchmark's grading protocol.
• intelligence per watt (IPW) (primary) — range: other
  • Accuracy (%) divided by energy consumption in joules per query. Normalizes performance by end-to-end inference energy cost.
• cost per query (USD) — range: USD
  • Calculated using token pricing: (input_tokens * input_cost_per_1M + output_tokens * output_cost_per_1M) / 1,000,000.

Input / output format

Input: Single-turn text queries (chat or reasoning prompts).

Output: Text response generated by the model.

Scoring recipe

def compute_metrics(predictions, gold, energy_joules_per_query, pricing):
    correct = sum(1 for p, g in zip(predictions, gold) if is_correct(p, g))
    accuracy = (correct / len(predictions)) * 100
    ipw = accuracy / energy_joules_per_query
    input_tokens = count_tokens(predictions, mode='input')
    output_tokens = count_tokens(predictions, mode='output')
    cost = (input_tokens * pricing['input'] + output_tokens * pricing['output']) / 1_000_000
    return {'accuracy': accuracy, 'ipw': ipw, 'cost': cost}

Common pitfalls

• Energy measurements must capture end-to-end inference costs on specific hardware configurations (e.g., PCIe vs SXM, HBM3e vs standard); FLOPs or theoretical power draw are insufficient.
• IPW normalization by 'joules per query' can be skewed by varying sequence lengths, as longer outputs consume more energy without proportionally increasing accuracy.
• Accuracy grading protocols differ across benchmarks (exact match vs. LLM-as-judge), which affects cross-dataset comparability and GDP-weighted calculations.

Evidence (verbatim from paper)

We measure both perplexity (left panel) and accuracy (right panel) normalized by energy consumption in joules per query, tracking nine distinct model families (Llama, Phi, Gemma, Mistral, Falcon, DeepSeek, Qwen, and GPT-OSS) deployed on various GPU configurations including NVIDIA A100 (40GB/80GB PCIe/SXM), H100 (80GB SXM), H200 (141GB HBM3e), and L40S (48GB) accelerators. Energy measurements capture end-to-end inference costs.

Citation

@misc{saadfalcon2025intelligence,
  title={Intelligence per Watt: Measuring Intelligence Efficiency of Local AI},
  author={Jon Saad-Falcon et al.},
  year={2025},
  note={arXiv:2511.07885}
}

• arXiv: 2511.07885