Futurepedia Eval

futurepedia-eval

Not All Languages are Equal: Insights into Multilingual Retrieval-Augmented Generation — Suhang Wu et al. (arXiv:2410.21970, 2024)

What this evaluates

This benchmark evaluates multilingual Retrieval-Augmented Generation (RAG) systems across three tasks: monolingual knowledge extraction, cross-lingual knowledge transfer, and multilingual knowledge selection. It probes a model's ability to retrieve and generate answers in eight languages, assess cross-lingual transfer capabilities, and measure selection bias when presented with conflicting answers across languages.

Datasets

• Futurepedia — total 197; splits: test (197); repo https://github.com/H-shw/futurepedia

Metrics

• Character 3-gram Recall (primary) — range: [0, 1]
  • Measures the proportion of character 3-grams from the ground truth answer that appear in the model's prediction. It is used instead of exact-match accuracy to handle diverse multilingual expressions.
• Selection Entropy (SE) — range: [0, 1]
  • Evaluates selection bias across languages. Calculated as SE = -Σ p(i) log(p(i)), where p(i) = f(i) / Σ f(j), and f(i) is the Character 3-gram Recall for the answer from the i-th language. Higher entropy indicates less bias.

Input / output format

Input: A question and one or more retrieved documents in one or multiple languages (English, French, Spanish, Portuguese, Chinese, Japanese, Korean, Arabic), depending on the evaluation task.

Output: A natural language answer string corresponding to the question.

Scoring recipe

def char_3gram_recall(pred, gold):
    gold_ngrams = {gold[i:i+3] for i in range(len(gold)-2)}
    pred_ngrams = {pred[i:i+3] for i in range(len(pred)-2)}
    if not gold_ngrams: return 0.0
    return len(gold_ngrams & pred_ngrams) / len(gold_ngrams)

def selection_entropy(recall_scores):
    total = sum(recall_scores)
    if total == 0: return 0.0
    probs = [r / total for r in recall_scores]
    return -sum(p * math.log(p) for p in probs if p > 0)

Common pitfalls

• Using standard exact-match accuracy instead of Character 3-gram Recall, which fails to capture valid paraphrases and diverse expressions in multilingual RAG.
• Misinterpreting Selection Entropy: higher values indicate lower selection bias (more uniform language selection), while lower values indicate strong bias toward specific languages.
• Reporting only average recall (AVG) without variance (VAR) masks significant performance disparities across languages.

Evidence (verbatim from paper)

The common practices of RAG often use Accuracy to evaluate whether the ground truth answer is fully contained in the prediction (Lewis et al. [2020]; Chen et al. [2023a]; Saad-Falcon et al. [2024]). However, as analyzed in (Chirkova et al. [2024]), one answer may have diverse expressions in multilingual RAG, and thus Accuracy fails to capture similarity in such cases. To deal with this issue, Chirkova et al. ([2024]) propose Character 3-gram Recall, which measures the proportion of 3-grams of ground truth answers that appear in the predictions. In this work, we use Character 3-gram Recall as our primary evaluation metric.

Citation

@misc{wu2024notalllanguages,
  title={Not All Languages are Equal: Insights into Multilingual Retrieval-Augmented Generation},
  author={Suhang Wu et al.},
  year={2024},
  note={arXiv:2410.21970}
}

• arXiv: 2410.21970