Multidomain RAG Eval

multidomain-rag-eval

Adapting Large Language Models for Multi-Domain Retrieval-Augmented-Generation — Misrahi et al. (2025) (arXiv:2504.02411, 2025)

What this evaluates

This benchmark evaluates the out-of-domain generalization and robustness of Retrieval-Augmented Generation (RAG) systems across diverse domains, answer formats, and context-criticality levels. It probes whether models can correctly extract and synthesize information from noisy or specialized document collections when internal knowledge is insufficient.

Datasets

• BioASQ — total ?; splits: test (-1)
• CovidQA — total ?; splits: test (-1)
• SearchQA — total ?; splits: test (-1)
• ParaphraseRC — total ?; splits: test (-1)
• SyllabusQA — total ?; splits: test (-1)
• TechQA — total ?; splits: test (-1)
• RobustQA — total ?; splits: test (-1)

Metrics

• LLMEval (primary) — range: [0, 1]
  • An open-source LLM is prompted with the question, context, generated response, and ground truth labels to output a binary judgment of correctness. The metric reports the fraction of responses judged correct.
• Match — range: [0, 1]
  • Binary metric that returns 1 if any ground truth label appears as a verbatim substring in the generated response, else 0.
• Recall — range: percent
  • Percentage of words from the ground truth labels that appear verbatim in the generated response.

Input / output format

Input: Query/question string and retrieved/reranked document chunks (context).

Output: Generated text response.

Scoring recipe

def score_llmeval(predictions, questions, contexts, ground_truths, evaluator_llm):
    correct = 0
    for pred, q, ctx, gt in zip(predictions, questions, contexts, ground_truths):
        prompt = f'Q: {q}\nCtx: {ctx}\nPred: {pred}\nGT: {gt}\nCorrect? (Yes/No)'
        if evaluator_llm.generate(prompt).strip().lower() == 'yes':
            correct += 1
    return correct / len(predictions)

def score_match(predictions, ground_truths):
    return sum(1 for p, g in zip(predictions, ground_truths) if any(label in p for label in g)) / len(predictions)

def score_recall(predictions, ground_truths):
    total = 0
    for p, g in zip(predictions, ground_truths):
        gt_w, pred_w = set(g.split()), set(p.split())
        total += len(gt_w & pred_w) / len(gt_w) if gt_w else 0
    return total / len(predictions)

Common pitfalls

• Match and Recall metrics are unsuitable for long-form QA tasks; Match often yields zero, and Recall is heavily skewed by common words.
• Evaluating out-of-domain generalization requires zero-shot inference; fine-tuning on the target domain masks the model's true robustness to domain shifts.
• Retrieval noise and overlapping context chunks can mislead generators, especially in context-critical domains like SyllabusQA.

Evidence (verbatim from paper)

To evaluate generated responses, we mostly use LLM evaluation, denoted as LLMEval, but we also consider Match and Recall. LLMEval prompts an open-source LLM to output a binary judgment about the correctness of the generated response, given the input question and the ground truth labels. ... LLMEval is particularly useful for comparing long generations/ground truth answers, since Match will always output the zero evaluation result, and Recall is highly impacted by common words, and hard to interepret.

Citation

@misc{misrahi2025adapting,
  title={Adapting Large Language Models for Multi-Domain Retrieval-Augmented-Generation},
  author={Misrahi et al. (2025)},
  year={2025},
  note={arXiv:2504.02411}
}

• arXiv: 2504.02411