qhjqhj00 inspiration-retrieval-eval
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition — Liu et al. (2025) (arXiv:2503.21248, 2025)
What this evaluates
Evaluates an LLM's ability to retrieve relevant prior research papers (inspirations) that can inform a given research question from a candidate pool. It measures how well models can surface novel, non-obvious knowledge links through iterative group-based selection.
Datasets
- ResearchBench Inspiration Retrieval — total 1386; splits: test (1386)
Metrics
Hit Ratio(primary) — range: percent- Number of groundtruth inspiration papers selected by the LLM divided by the total number of inspiration candidates. Evaluated at top 20% (15/75) and top 4% (3/75) retention rates after iterative group selection.
Input / output format
Input: Research question, groundtruth inspirations, negative inspirations, and an inspiration candidate set of 75 papers (each represented by title and abstract). The set is iteratively split into groups of 15, and the LLM selects top 3 per group across rounds.
Output: A list of selected paper IDs from the candidate set.
Scoring recipe
def hit_ratio(predictions, gold):
hits = sum(1 for p in predictions if p in gold)
return hits / len(gold)
Common pitfalls
- Iterative group-based selection process differs from standard top-k retrieval, requiring multiple rounds of filtering.
- Position bias and training data co-occurrence can inflate retrieval of statistically closer papers, skewing results.
Evidence (verbatim from paper)
We use Hit Ratio as the evaluation metric, it is calculated as the number of groundtruth inspiration papers selected by the LLM divided by the total number of inspiration candidates.
Citation
@misc{liu2025researchbench,
title={ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition},
author={Liu et al. (2025)},
year={2025},
note={arXiv:2503.21248}
}
- arXiv: 2503.21248