Mtrag Un Eval

mtrag-un-eval

MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations — Rosenthal et al. (2026) (arXiv:2602.23184, 2026)

What this evaluates

Evaluates multi-turn RAG systems on handling unanswerable, underspecified, and non-standalone questions. It probes both retrieval ranking quality and generation quality, including the model's ability to correctly refuse or request clarification when context is insufficient.

Datasets

• MTRAG-UN — total 666; splits: test (-1); repo https://github.com/IBM/mt-rag-benchmark

Metrics

• RB_llm (primary) — range: [0, 1]
  • LLM-as-a-judge metric where a reference LLM (GPT-OSS-120B) scores the generated response against the ground truth answer and provided context. Scores are conditioned on answerability.
• nDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5, measuring the quality of the ranked list of retrieved documents relative to an ideal ranking.

Input / output format

Input: Retrieval: conversational history + current turn query. Generation: retrieved passages (top 5 or up to 10) + user question + system prompt instructing <150 word response, IDK fallback, and clarification handling.

Output: Retrieval: ranked list of document IDs/text. Generation: natural language response (<150 words), optionally containing "I do not have specific information" or a clarification request.

Scoring recipe

def score_rb_llm(predictions, gold_answers, contexts, judge_model):
    scores = []
    for pred, gold, ctx in zip(predictions, gold_answers, contexts):
        # Judge model evaluates pred against gold and ctx
        score = judge_model.evaluate(response=pred, reference=gold, context=ctx)
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Models frequently fail to recognize unanswerable questions, instead generating plausible but hallucinated answers.
• Query rewrite significantly impacts non-standalone question performance, but retrieval drops sharply in Banking/Telco domains due to dense hyperlinks and cross-page dependencies.
• LLM judges (GPT-OSS-120B) are used as proxies for human judgment, but their correlation and potential biases should be considered when interpreting generation scores.

Evidence (verbatim from paper)

We adopt the evaluation metrics of Katsis et al. (2025): (1) reference-based $ extrm{RB}{ extrm{llm}}$ and $ extrm{RB}{ extrm{alg}}$, (2) the IDK ("I Don’t Know") judge, and (3) faithfulness judge from RAGAS RL${}_{ extrm{F}}$. All evaluation metrics are conditioned to account for answerability.

Citation

@misc{rosenthal2026mtragun,
  title={MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations},
  author={Rosenthal et al. (2026)},
  year={2026},
  note={arXiv:2602.23184}
}

• arXiv: 2602.23184