Open Domain QA Eval

open-domain-qa-eval

Optimizing open-domain question answering with graph-based retrieval augmented generation — Cahoon et al. (2025) (arXiv:2503.02922, 2025)

What this evaluates

Evaluates open-domain question answering systems on their ability to retrieve relevant context and generate accurate answers across straightforward (OLTP) and synthesis-heavy (OLAP) queries. It measures factual correctness against reference answers and assesses multi-dimensional answer quality (comprehensiveness, diversity, empowerment) for open-ended questions.

Datasets

• HotPotQA — total 5491; splits: dev (5491)
• MSMarco — total 1000; splits: test (1000)
• Microsoft Earnings Call Transcripts — total 40; splits: test (40)
• Kevin Scott Podcast Transcripts — total 125; splits: test (125)

Metrics

• LLM-as-a-judge accuracy (primary) — range: [0, 1]
  • Binary correctness score for OLTP queries where the LLM judge outputs YES (correct) or NO (incorrect) by comparing the generated answer to a reference answer. For OLAP queries, it aggregates qualitative scores across Comprehensiveness, Diversity, and Empowerment dimensions.

Input / output format

Input: Open-domain question/query and retrieved context (from graph, vector, or hybrid retrieval pipelines).

Output: Generated natural language answer. For evaluation, the LLM judge outputs a binary YES/NO for OLTP or qualitative scores for OLAP dimensions.

Scoring recipe

def evaluate(predictions, golds, query_type):
    if query_type == 'OLTP':
        correct = [1 if judge(prediction, gold) == 'YES' else 0 for prediction, gold in zip(predictions, golds)]
        return sum(correct) / len(correct)
    else:  # OLAP
        scores = [judge_comprehensive(p) + judge_diverse(p) + judge_empower(p) for p in predictions]
        return sum(scores) / len(scores)

Common pitfalls

• OLAP queries lack fixed ground truth, making correctness subjective and requiring multi-dimensional scoring rather than exact match.
• LLM-as-a-judge introduces variability and potential bias compared to human experts, despite reported 80%+ agreement.
• HotPotQA requires filtering out queries directly answerable without context to prevent data leakage and ensure retrieval is actually tested.

Evidence (verbatim from paper)

To assess answer quality, we apply LLM-as-a-judge for both OLTP and OLAP-style benchmarks: OLTP Evaluation: Since answers have clear ground truths, we compare model-generated responses against reference answers, achieving 99%+ agreement. To ensure consistency, we apply a logit bias, restricting outputs to “YES” (correct) or “NO” (incorrect). OLAP Evaluation: As no fixed ground truth exists, we adopt the GraphRAG evaluation framework, assessing responses on: Comprehensiveness: Depth and thoroughness of information. Diversity: Inclusion of multiple perspectives. Empowerment: How well the answer informs decision-making.

Citation

@misc{cahoon2025optimizing,
  title={Optimizing open-domain question answering with graph-based retrieval augmented generation},
  author={Cahoon et al. (2025)},
  year={2025},
  note={arXiv:2503.02922}
}

• arXiv: 2503.02922