Simulrag Scientific QA Eval

simulrag-scientific-qa-eval

SimulRAG: Simulator-based RAG for Grounding LLMs in Long-form Scientific QA — Xu et al. (2025) (arXiv:2509.25459, 2025)

What this evaluates

Evaluates long-form scientific question answering by measuring how effectively a model generates informative and factual answers using simulator-based retrieval. It also assesses the efficiency and quality of claim-level verification and updating strategies under varying computational budgets.

Datasets

• Climate modeling dataset — total 200; splits: test (200)
• Epidemiological modeling dataset — total 200; splits: test (200)

Metrics

• informativeness — range: count
  • Counts the number of unique true claims within the generated answer.
• factuality (primary) — range: [0, 1]
  • Measures the proportion of true claims across all generated claims. Calculated as (number of true claims) / (total number of claims).
• F1 — range: [0, 1]
  • Balanced F1 score used to identify thresholds where precision approximates recall for updated claim sets based on uncertainty estimation scores.
• AUPR — range: [0, 1]
  • Area Under the Precision-Recall curve computed for updated claim sets using uncertainty estimation scores.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve computed for updated claim sets using uncertainty estimation scores.

Input / output format

Input: Free-form scientific questions covering climate phenomena or disease spread dynamics.

Output: Long-form scientific answers decomposed into atomic claims, each evaluated for correctness and relevance to the posed question.

Scoring recipe

claims = decompose(answer)
verified = []
for c in claims:
    if is_correct(c, ground_truth) and is_relevant(c, question):
        verified.append(True)
    else:
        verified.append(False)
informativeness = count_unique_true_claims(verified)
factuality = sum(verified) / len(verified)
# For claim-level methods, compute F1, AUPR, AUROC
# by comparing verification scores against verified labels

Common pitfalls

• Claims are only labeled true if they are both factually correct AND relevant to the posed question; irrelevant true facts are marked false.
• Claim decomposition relies on structured LLM prompting, which can introduce splitting errors that affect downstream verification.
• Baselines require manual adaptation to extract textual context from simulation outputs, which may not perfectly replicate native simulator interfaces.

Evidence (verbatim from paper)

Informativeness counts unique true claims within answers. Factuality measures the proportion of true claims across all generated claims. Claims receive true labels only when both correct and relevant to the posed question.

Citation

@misc{xu2025simulrag,
  title={SimulRAG: Simulator-based RAG for Grounding LLMs in Long-form Scientific QA},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2509.25459}
}

• arXiv: 2509.25459