Xrag Eval

xrag-eval

XRAG: Cross-lingual Retrieval-Augmented Generation — Liu et al. (2025) (arXiv:2505.10089, 2025)

What this evaluates

Evaluates an LLM's ability to perform cross-lingual retrieval-augmented generation by answering questions in a target language using supporting documents in English or mixed languages, while ignoring topically related distractors. It specifically probes cross-document reasoning capabilities and response language consistency.

Datasets

• XRAG — total ?; splits: test (-1)

Metrics

• response language consistency (primary) — range: percent
  • Measures whether the generated answer is in the same language as the input question. Calculated as the proportion of instances where Language(q) == Language(\tilde{a}).

Input / output format

Input: A question q in a target language (German, Spanish, Chinese, or Arabic), a set of two supporting articles D^+, and a set of distracting articles D^- (topically related but unhelpful). In the monolingual retrieval setting, all documents are in English. In the multilingual retrieval setting, documents are in English and the question language.

Output: An answer \tilde{a} generated by the LLM, which must be in the same language as the question q.

Scoring recipe

def score_response_language_consistency(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        if detect_language(pred) == detect_language(gold_ans):
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Models frequently generate answers in English even when the question is in a non-English language, failing the language consistency constraint in monolingual retrieval settings.
• Distracting articles D^- are topically related to the question but cannot answer it, requiring the model to perform cross-document reasoning to correctly ignore them and synthesize information only from D^+.

Evidence (verbatim from paper)

The dataset reveals two key challenges: (1) all evaluated LLMs fail to maintain response language consistency in monolingual retrieval, and (2) in multilingual retrieval, the core difficulty lies in cross-lingual information integration rather than non-English text generation.

Citation

@misc{liu2025xrag,
  title={XRAG: Cross-lingual Retrieval-Augmented Generation},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2505.10089}
}

• arXiv: 2505.10089