qhjqhj00 hypothesis-composition-eval
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition — Liu et al. (2025) (arXiv:2503.21248, 2025)
What this evaluates
Assesses an LLM's capability to synthesize novel research hypotheses by combining a given research background with retrieved inspiration papers. It probes the model's ability to mutate and recombine scientific concepts into coherent, groundtruth-aligned proposals.
Datasets
- ResearchBench Hypothesis Composition — total 1386; splits: test (1386)
Metrics
Normalized Composition Score(primary) — range: [0, 1]- Average score on a 0-5 Likert scale measuring coverage of key points in the groundtruth hypothesis, normalized by dividing by the maximum possible score (5).
Input / output format
Input: Research background, groundtruth inspiration papers (2-3), and prompts for mutate/refine/recombine steps.
Output: A composed research hypothesis text.
Scoring recipe
def normalized_score(predictions, gold):
raw = llm_judge_score(predictions, gold) # 0 to 5
return raw / 5.0
Common pitfalls
- Scoring relies on a 6-point Likert scale judged by LLM/experts, which may vary across runs or judges.
- Task uses groundtruth inspirations for evaluation, not retrieved ones, isolating composition ability from retrieval errors.
Evidence (verbatim from paper)
we use a 6-point Likert scale (from 0 to 5) to measure whether the generated hypothesis has covered the key points in the groundtruth hypothesis. To compute the generation accuracy, we normalize the average score by dividing it by the maximum possible score (5).
Citation
@misc{liu2025researchbench,
title={ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition},
author={Liu et al. (2025)},
year={2025},
note={arXiv:2503.21248}
}
- arXiv: 2503.21248