Xtreme R Eval

xtreme-r-eval

XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation — Ruder et al. (2021) (arXiv:2104.07412, 2021)

What this evaluates

Evaluates zero-shot cross-lingual transfer by training models on English data and testing them on 50 typologically diverse languages across classification, QA, and retrieval tasks. It probes fine-grained diagnostic capabilities and cross-lingual alignment using structured performance breakdowns.

Datasets

• XQuAD — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/google-research/xtreme
• XCOPA — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/google-research/xtreme
• Mewsli-X — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/google-research/xtreme
• LAReQA — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/google-research/xtreme
• CheckList — total ?; splits: test (-1); repo https://github.com/google-research/xtreme

Metrics

• Exact Match (primary) — range: [0, 1]
  • Returns 1.0 if the predicted answer string exactly matches the gold answer string, and 0.0 otherwise. Averaged across all test instances.
• F1 — range: [0, 1]
  • Token-level F1 score measuring the harmonic mean of precision and recall between predicted and gold answers.

Input / output format

Input: Training: English-language task instances. Evaluation: Target-language test instances (questions, multiple-choice options, or document pairs) for zero-shot inference.

Output: Predicted answer string, class label, or retrieved document ID per instance.

Scoring recipe

def evaluate(predictions, golds):
    em = sum(1.0 if p == g else 0.0 for p, g in zip(predictions, golds)) / len(golds)
    f1 = compute_token_f1(predictions, golds)
    return {'exact_match': em, 'f1': f1, 'avg': (em + f1) / 2}

Common pitfalls

• Using validation sets from multiple target languages for hyperparameter tuning instead of a single target language validation set as recommended.
• Fine-tuning on translated in-language data rather than strictly following the zero-shot cross-lingual transfer protocol.
• Reporting only aggregate scores without providing fine-grained diagnostic breakdowns, which obscures language-specific or attribute-specific weaknesses.

Evidence (verbatim from paper)

We average “F1” and “Exact Match” of QA systems. We classify the attribute values into four categories: extra-small (XS), small (S), large (L) and extra-large (XL) values (see the Appendix for detailed interval information).

Citation

@misc{ruder2021xtremer,
  title={XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation},
  author={Ruder et al. (2021)},
  year={2021},
  note={arXiv:2104.07412}
}

• arXiv: 2104.07412