Mlqa Xquad Eval

mlqa-xquad-eval

Improving Low-resource Reading Comprehension via Cross-lingual Transposition Rethinking — Gaochen Wu et al. (2021) (arXiv:2107.05002, 2021)

What this evaluates

This benchmark evaluates a model's ability to perform cross-lingual extractive reading comprehension in a zero-shot setting. It probes how well a model trained on high-resource English (and Chinese) data can generalize to answer questions in low-resource languages by leveraging translated parallel corpora and multilingual attention mechanisms.

Datasets

• MLQA — total 46444; splits: dev (4199), test (42245)
• XQuAD — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of token-level precision and recall over the predicted and gold answer spans. Scores are averaged first within each target language, then the language-level averages are combined to report the final score.
• EM — range: percent
  • Binary score (1 if predicted answer exactly matches gold answer span, 0 otherwise). Averaged first within each target language, then across languages.

Input / output format

Input: A context paragraph and a question, both in the target evaluation language.

Output: A text span extracted verbatim from the context paragraph that answers the question.

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores = []
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        pred_tokens = normalize(pred)  # strip punctuation/articles
        gold_tokens = normalize(gold)
        em_scores.append(1.0 if pred_tokens == gold_tokens else 0.0)
        if not gold_tokens: continue
        prec = len(set(pred_tokens) & set(gold_tokens)) / len(pred_tokens)
        rec = len(set(pred_tokens) & set(gold_tokens)) / len(gold_tokens)
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return {'EM': mean(em_scores), 'F1': mean(f1_scores)}

Common pitfalls

• The paper explicitly requires answer preprocessing (stripping Unicode punctuation and standalone articles) before computing metrics, which is often overlooked.
• Metrics are averaged within each target language first, and then the language-level averages are combined to report the final score, rather than averaging globally across all instances.
• Evaluation is strictly zero-shot: models are trained only on English (and Chinese) data and evaluated directly on translated target-language test sets without fine-tuning on target-language data.

Evidence (verbatim from paper)

As far as we know, most of extractive RC tasks adopt EM and F1 metrics as evaluation measures. If prediction answer exactly matches the golden answer span, EM is equal to 1, otherwise 0. As for F1, a harmonic mean of precision and recall, each of which is calculated over instances within a language. The EM and F1 scores for each instance are averaged within each target language to obtain final F1 and EM scores, respectively. We take advantage of the multilingual evaluation script for the MLQA and XQuAD tasks to perform answer preprocessing operations such as stripping Unicode punctuation, standalone article stripping.

Citation

@misc{wu2021improving,
  title={Improving Low-resource Reading Comprehension via Cross-lingual Transposition Rethinking},
  author={Gaochen Wu et al. (2021)},
  year={2021},
  note={arXiv:2107.05002}
}

• arXiv: 2107.05002