Xtreme Eval

xtreme-eval

XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization — Junjie Hu et al. (2020) (arXiv:2003.11080, 2020)

What this evaluates

Evaluates zero-shot cross-lingual transfer of multilingual language models. Models are trained exclusively on English-labeled data and then tested on 40 typologically diverse languages across nine tasks spanning sentence classification, structured prediction, question answering, and sentence retrieval.

Datasets

• XTREME — total ?; splits: train (-1), test (-1); repo https://github.com/google-research/xtreme

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of total instances.
• F1 — range: [0, 1]
  • Macro-averaged F1 score, computed as the harmonic mean of precision and recall across all classes or tokens.
• exact-match (EM) — range: [0, 1]
  • Exact match for question answering: 1 if the predicted answer span exactly matches the gold answer span, 0 otherwise.

Input / output format

Input: Multilingual text sequences (sentences, paragraphs, or QA pairs) provided in English for training and in 40 target languages for testing.

Output: Task-specific predictions: class labels for XNLI/PAWS-X/Tatoeba, token tags for POS/NER, answer spans for XQuAD/MLQA/TyDiQA-GoldP, and relevance scores for BUCC.

Scoring recipe

def evaluate(predictions, gold, task):
    if task in ['XNLI', 'PAWS-X', 'Tatoeba']:
        return accuracy(predictions, gold)
    elif task in ['POS', 'NER', 'BUCC']:
        return f1_score(predictions, gold, average='macro')
    elif task in ['XQuAD', 'MLQA', 'TyDiQA-GoldP']:
        em = exact_match(predictions, gold)
        f1 = qa_f1(predictions, gold)
        return f1, em
    return None

Common pitfalls

• Using target-language training data beyond the provided English labels violates the strict zero-shot cross-lingual transfer constraint.
• Directly comparing scores across languages for tasks with different test sets (POS, NER, MLQA, TyDiQA-GoldP) is invalid as noted by the authors.
• Failing to properly align answer spans when translating QA data for translate-train baselines leads to incomparable scores.

Evidence (verbatim from paper)

Concretely, pretrained multilingual representations are fine-tuned on English labelled data of an xtreme task. The model is then evaluated on the test data of the task in the target languages. Metrics: Acc., F1, F1/EM, F1, Acc.

Citation

@misc{hu2020xtreme,
  title={XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization},
  author={Junjie Hu et al. (2020)},
  year={2020},
  note={arXiv:2003.11080}
}

• arXiv: 2003.11080