qhjqhj00 entropy-minimization-reasoning-eval
The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning — Agarwal et al. (2025) (arXiv:2505.15134, 2025)
What this evaluates
Evaluates the reasoning capabilities of LLMs on mathematical and coding benchmarks by measuring accuracy under various inference-time scaling and unsupervised entropy minimization techniques. It probes whether reducing output uncertainty improves correctness without labeled data or parameter updates.
Datasets
- AMC — total ?; splits: test (-1)
- AIME — total ?; splits: test (-1)
- Minerva — total ?; splits: test (-1)
- LeetCode Live Contest — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly solved problems out of the total number of evaluated instances. Calculated as (correct predictions / total predictions) * 100.
Input / output format
Input: Natural language prompts containing mathematical problems or coding tasks requiring code generation.
Output: Model-generated text containing either a mathematical answer or executable code.
Scoring recipe
correct = 0
for pred, gold in zip(predictions, golds):
if check_correctness(pred, gold):
correct += 1
return (correct / len(golds)) * 100
Common pitfalls
- EM-INF reduces output diversity, which can cause performance gains to diminish when combined with iterative self-refinement methods that rely on diverse generations.
- Accuracy metrics are reported across heterogeneous benchmarks (math vs. coding) with different difficulty levels, making direct averaging potentially misleading without stratification.
Evidence (verbatim from paper)
Accuracy vs. FLOPs for combining EM-INF and self-consistency at inference time on AMC. As expected, EM-INF reduces the diversity of the generation, which causes it’s benefits to diminish over iterations.
Citation
@misc{agarwal2025entropyminimization,
title={The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning},
author={Agarwal et al. (2025)},
year={2025},
note={arXiv:2505.15134}
}
- arXiv: 2505.15134