Scholarqa Bench Eval

scholarqa-bench-eval

OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs — Asai et al. (2024) (arXiv:2411.14199, 2024)

What this evaluates

Evaluates LLMs' ability to synthesize scientific literature by answering open-ended, multi-domain questions using retrieved papers. It probes long-form generation, factual correctness, citation accuracy, and content quality/organization across single- and multi-paper retrieval setups.

Datasets

• ScholarQABench — total 5393; splits: test (5393)

Metrics

• Corr — range: [0, 1]
  • Accuracy for binary tasks (SciFact, PubMedQA), ROUGE-L for QASA, and a weighted LLM score (GPT-4o-turbo) based on expert-annotated rubrics (40% general, 60% annotation-driven) for ScholarQA-CS.
• Citation F1 (primary) — range: [0, 1]
  • Harmonic mean of Citation Precision (relevance and necessity of citations) and Citation Recall (coverage of citation-worthy statements). Evaluated via LLM verification of whether citations support statements and if their removal impacts integrity.
• LLM-Score — range: [1, 5]
  • Prometheus v2 assigns 1-5 scale scores for Relevance, Coverage, and Organization based on defined rubrics. Human evaluation also measures Overall Usefulness.

Input / output format

Input: Open-ended scientific question requiring multi-paper retrieval, accompanied by retrieved passages/papers from a large open-access datastore.

Output: Long-form natural language response with inline reference numbers (e.g., [1], [2]) linked to the provided passages.

Scoring recipe

def score_correctness(pred, gold, task):
    if task in ['SciFact', 'PubMedQA']:
        return 1.0 if pred.strip().lower() == gold.strip().lower() else 0.0
    if task == 'QASA':
        return rouge_l_score(pred, gold)
    if task == 'ScholarQA-CS':
        rubric = load_rubric(gold)
        return gpt4o_score(pred, rubric)
    return 0.0

def score_citation_f1(pred):
    statements = extract_statements(pred)
    citations = extract_citations(pred)
    recall = sum(1 for s in statements if has_appropriate_citation(s, citations)) / len(statements)
    precision = sum(1 for c in citations if is_relevant_and_necessary(c, pred)) / len(citations)
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0

Common pitfalls

• LLM-based evaluation of long-form answers can be biased by response length or formatting rather than factual accuracy.
• Citation precision/recall relies on LLMs to judge 'necessity' and 'support', which may over-penalize or under-penalize implicit citations.
• Rubric-based scoring for ScholarQA-CS is subjective; annotator agreement varies significantly (59.5–79.3 Pearson correlation).

Evidence (verbatim from paper)

For each citation, we then verify its relevance and necessity—specifically, whether the citation supports the statement and if its removal impacts the integrity of remaining citations (Citation Precision, -p). Finally, we compute Citation F1 ( -F1) as well, and use it as a primarily metric for citation accuracy.

Citation

@misc{asai2024openscholar,
  title={OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs},
  author={Asai et al. (2024)},
  year={2024},
  note={arXiv:2411.14199}
}

• arXiv: 2411.14199