# Futurepedia Eval

> futurepedia-eval

- Skill: `qhjqhj00/futurepedia-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/futurepedia-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/futurepedia-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/futurepedia-eval

---


# futurepedia-eval

> Not All Languages are Equal: Insights into Multilingual Retrieval-Augmented Generation — Suhang Wu et al. (arXiv:2410.21970, 2024)

## What this evaluates

This benchmark evaluates multilingual Retrieval-Augmented Generation (RAG) systems across three tasks: monolingual knowledge extraction, cross-lingual knowledge transfer, and multilingual knowledge selection. It probes a model's ability to retrieve and generate answers in eight languages, assess cross-lingual transfer capabilities, and measure selection bias when presented with conflicting answers across languages.

## Datasets

- **Futurepedia** — total 197; splits: test (197); repo https://github.com/H-shw/futurepedia

## Metrics

- `Character 3-gram Recall` **(primary)** — range: [0, 1]
  - Measures the proportion of character 3-grams from the ground truth answer that appear in the model's prediction. It is used instead of exact-match accuracy to handle diverse multilingual expressions.
- `Selection Entropy (SE)` — range: [0, 1]
  - Evaluates selection bias across languages. Calculated as SE = -Σ p(i) log(p(i)), where p(i) = f(i) / Σ f(j), and f(i) is the Character 3-gram Recall for the answer from the i-th language. Higher entropy indicates less bias.

## Input / output format

**Input**: A question and one or more retrieved documents in one or multiple languages (English, French, Spanish, Portuguese, Chinese, Japanese, Korean, Arabic), depending on the evaluation task.

**Output**: A natural language answer string corresponding to the question.

## Scoring recipe

```python
def char_3gram_recall(pred, gold):
    gold_ngrams = {gold[i:i+3] for i in range(len(gold)-2)}
    pred_ngrams = {pred[i:i+3] for i in range(len(pred)-2)}
    if not gold_ngrams: return 0.0
    return len(gold_ngrams & pred_ngrams) / len(gold_ngrams)

def selection_entropy(recall_scores):
    total = sum(recall_scores)
    if total == 0: return 0.0
    probs = [r / total for r in recall_scores]
    return -sum(p * math.log(p) for p in probs if p > 0)
```

## Common pitfalls

- Using standard exact-match accuracy instead of Character 3-gram Recall, which fails to capture valid paraphrases and diverse expressions in multilingual RAG.
- Misinterpreting Selection Entropy: higher values indicate lower selection bias (more uniform language selection), while lower values indicate strong bias toward specific languages.
- Reporting only average recall (AVG) without variance (VAR) masks significant performance disparities across languages.

## Evidence (verbatim from paper)

> The common practices of RAG often use Accuracy to evaluate whether the ground truth answer is fully contained in the prediction *(Lewis et al. [2020]; Chen et al. [2023a]; Saad-Falcon et al. [2024])*. However, as analyzed in *(Chirkova et al. [2024])*, one answer may have diverse expressions in multilingual RAG, and thus Accuracy fails to capture similarity in such cases. To deal with this issue, *Chirkova et al. ([2024])* propose Character 3-gram Recall, which measures the proportion of 3-grams of ground truth answers that appear in the predictions. In this work, we use Character 3-gram Recall as our primary evaluation metric.

## Citation

```bibtex
@misc{wu2024notalllanguages,
  title={Not All Languages are Equal: Insights into Multilingual Retrieval-Augmented Generation},
  author={Suhang Wu et al.},
  year={2024},
  note={arXiv:2410.21970}
}
```

- arXiv: 2410.21970