Xglue Eval

xglue-eval

XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation — Liang et al. (2020) (arXiv:2004.01401, 2020)

What this evaluates

Evaluates cross-lingual transfer capabilities of pre-trained language models across 11 diverse natural language understanding and generation tasks spanning over 100 languages. It measures how well models fine-tuned on English can generalize to zero-shot testing in other languages.

Datasets

• XGLUE — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted labels averaged across all target languages for understanding tasks (NER, POS, NC, XNLI, PAWS-X, QADSM, WPR, QAM).
• BLEU-4 — range: percent
  • 4-gram BLEU score for generation tasks (QG, NTG) and abstractive summarization.
• ROUGE-L — range: percent
  • Longest common subsequence recall for abstractive summarization evaluation.

Input / output format

Input: Source text(s) in a target language, optionally paired with a pivot language (English) training set for fine-tuning.

Output: Predicted class labels, token-level tags, or generated text sequences.

Scoring recipe

def compute_metric(predictions_by_lang, gold_by_lang, task_type):
    lang_scores = []
    for lang in predictions_by_lang:
        preds = predictions_by_lang[lang]
        gold = gold_by_lang[lang]
        if task_type == 'understanding':
            lang_scores.append(sum(1 for p, g in zip(preds, gold) if p == g) / len(gold))
        else:
            lang_scores.append(bleu4(preds, gold))
    return mean(lang_scores)

Common pitfalls

• Models are fine-tuned exclusively on English labeled data, then zero-shot tested on all other languages; failing to replicate this pivot-language setup yields non-comparable results.
• Generation tasks use different pre-training objectives (xDAE vs xFNP) and prediction mechanisms (single-token vs multi-token), making direct comparison unfair without noting the architectural differences.
• Averages reported in tables are macro-averages across languages, not weighted by dataset size.

Evidence (verbatim from paper)

Given a task, each pre-trained model is fine-tuned using its English training set only, and then applied to all test sets in different languages. ... BLUE-4 is the metric.

Citation

@misc{liang2020xglue,
  title={XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation},
  author={Liang et al. (2020)},
  year={2020},
  note={arXiv:2004.01401}
}

• arXiv: 2004.01401