Multilingual Transfer Eval

multilingual-transfer-eval

Unsupervised Cross-lingual Representation Learning at Scale — Conneau et al. (2019) (arXiv:1911.02116, 2019)

What this evaluates

Probes cross-lingual transfer and multilingual representation learning across natural language inference, named entity recognition, question answering, and English text classification. The protocol evaluates how well a model trained on English data generalizes to 14 other languages, while also measuring per-language and multilingual fine-tuning performance.

Datasets

• XNLI — total ?; splits: train (-1), dev (-1), test (-1)
• CoNLL-2002/2003 — total ?; splits: train (-1), dev (-1), test (-1)
• MLQA — total ?; splits: train (-1), dev (-1), test (-1)
• GLUE — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 × (precision × recall) / (precision + recall). Used for NER and MLQA.
• Exact Match (EM) — range: [0, 1]
  • Proportion of predictions that exactly match the gold answer string. Used for MLQA.
• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted class labels out of total instances. Used for XNLI and GLUE.

Input / output format

Input: For NLI and GLUE: sentence or sentence-pair text. For NER: tokenized word sequences. For QA: question and context paragraph text.

Output: For NLI/GLUE: discrete class label. For NER: sequence of BIO entity tags per token. For QA: predicted answer span text.

Scoring recipe

def compute_metrics(preds, golds, task):
    if task in ['NER', 'MLQA']:
        em = sum(p == g for p, g in zip(preds, golds)) / len(golds)
        tp = sum(1 for p, g in zip(preds, golds) if p == g)
        prec = tp / max(1, sum(1 for p in preds if p))
        rec = tp / max(1, sum(1 for g in golds if g))
        f1 = 2 * prec * rec / max(1e-9, prec + rec)
        return {'EM': em, 'F1': f1}
    else:  # XNLI, GLUE
        acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
        return {'Accuracy': acc}

Common pitfalls

• Cross-lingual transfer strictly uses only the English training set; fine-tuning on machine-translated data is a separate baseline.
• CoNLL NER uses token-level BIO tagging and reports token-level F1, not span-level F1.
• GLUE reports the average of task-specific metrics (accuracy or F1), not a single raw score.
• MLQA and XNLI dev/test sets contain multiple languages, but the transfer protocol evaluates English-to-other-language generalization only.

Evidence (verbatim from paper)

We report the F1 score, and compare to baselines from Lample et al. (2016) and Akbik et al. (2018). We report the F1 score as well as the exact match (EM) score for cross-lingual transfer from English.

Citation

@misc{conneau2019xlmr,
  title={Unsupervised Cross-lingual Representation Learning at Scale},
  author={Conneau et al. (2019)},
  year={2019},
  note={arXiv:1911.02116}
}

• arXiv: 1911.02116