Litqa2 Eval

litqa2-eval

Patience is all you need! An agentic system for performing scientific literature review — Brett et al. (2025) (arXiv:2504.08752, 2025)

What this evaluates

Evaluates an agentic system's ability to retrieve relevant scientific literature, re-rank passages, and answer multiple-choice questions based on the retrieved text. It probes retrieval coverage, passage localization, and question-answering accuracy under information loss constraints.

Datasets

• LitQA2 — total 199; splits: test (199)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Accuracy = Total correct answers / Total questions.
• Coverage — range: [0, 1]
  • Coverage = Total attempted answers / Total questions.
• Precision — range: [0, 1]
  • Precision = Total correct answers / Total attempted answers.

Input / output format

Input: A multiple-choice question with answer options, and a set of retrieved scientific articles or text chunks.

Output: A single selected answer option from the provided choices.

Scoring recipe

correct = sum(1 for p, g in zip(predictions, golds) if p == g)
attempted = len(predictions)
total = len(golds)
accuracy = correct / total
coverage = attempted / total
precision = correct / attempted

Common pitfalls

• Evaluation is restricted to the PubMed Central Open Access subset (103/199 questions) due to full-text availability constraints.
• The benchmark assumes a single source paper contains the answer, but the authors find this assumption often fails in their corpus.
• Metrics are reported with 95% confidence intervals over 3 repeats, not just point estimates.

Evidence (verbatim from paper)

For evaluating the retrieval on full text articles in the PubMed Central Open Access dataset we can only consider a subset of the LitQA2 benchmark*[[12]]* (for which only 103 out of 199 are found in the PubMed Central OA subset). ... In the assessment of the benchmark we follow the same conventions as in the original paper*[[12]]*: Accuracy = Total correct answers / Total questions, Coverage = Total attempted answers / Total questions, Precision = Total correct answers / Total attempted answers

Citation

@misc{brett2025patience,
  title={Patience is all you need! An agentic system for performing scientific literature review},
  author={Brett et al. (2025)},
  year={2025},
  note={arXiv:2504.08752}
}

• arXiv: 2504.08752