qhjqhj00 hypothesis-ranking-eval
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition — Liu et al. (2025) (arXiv:2503.21248, 2025)
What this evaluates
Measures an LLM's ability to correctly rank a groundtruth hypothesis against a set of negative hypotheses using pairwise comparisons. It evaluates discriminative judgment in scientific reasoning.
Datasets
- ResearchBench Hypothesis Ranking — total 1386; splits: test (1386)
Metrics
Accuracy(primary) — range: percent- Proportion of correct pairwise rankings out of 15 comparisons. Each pair is compared twice with reversed positions to mitigate position bias, and results are averaged.
Input / output format
Input: A set of 16 hypotheses (1 groundtruth, 15 negative) derived from background and inspirations. Presented as pairwise comparisons.
Output: A binary preference or ranking decision for each pair.
Scoring recipe
def ranking_accuracy(predictions, gold):
correct = sum(1 for p in predictions if p == gold)
return correct / 15.0
Common pitfalls
- Strong position bias affects many models; requires double-pass evaluation with reversed positions to average out.
- Negative hypotheses are generated via the same framework, not human-written, which may limit diversity.
Evidence (verbatim from paper)
Accuracy is used as the evaluation metric, which is calculated as the proportion of correct pairwise rankings out of 15 comparisons. During the pairwise evaluation, we find that many LLMs have strong position bias: they largely prefer the first hypothesis than the second. To avoid this bias, for each hypothesis pair, we compare them twice with reverse positions, and the results are averaged.
Citation
@misc{liu2025researchbench,
title={ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition},
author={Liu et al. (2025)},
year={2025},
note={arXiv:2503.21248}
}
- arXiv: 2503.21248