Mlqa Eval

mlqa-eval

MLQA: Evaluating Cross-lingual Extractive Question Answering — Lewis et al. (2019) (arXiv:1910.07475, 2019)

What this evaluates

Evaluates cross-lingual extractive question answering by measuring how well models can answer questions in one language using context in another, and how performance generalizes across different language pairs.

Datasets

• MLQA — total ?; splits: test (-1); repo https://github.com/facebookresearch/mlqa

Metrics

• F1 score (primary) — range: [0, 1]
  • Token-level F1 score measuring the overlap between the predicted answer span and the gold answer span.
• Exact Match — range: [0, 1]
  • Binary metric that scores 1 if the predicted answer span exactly matches the gold answer span, and 0 otherwise.

Input / output format

Input: A question in a target language and a context paragraph in either the same or a different language.

Output: A text span extracted verbatim from the provided context.

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores = []
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        em_scores.append(1.0 if pred == gold else 0.0)
        pred_tokens = set(pred.split())
        gold_tokens = set(gold.split())
        if not pred_tokens or not gold_tokens:
            f1_scores.append(0.0)
        else:
            common = pred_tokens & gold_tokens
            precision = len(common) / len(pred_tokens)
            recall = len(common) / len(gold_tokens)
            f1 = 2 * precision * recall / (precision + recall)
            f1_scores.append(f1)
    return sum(em_scores) / len(em_scores), sum(f1_scores) / len(f1_scores)

Common pitfalls

• MLQA uses a single gold answer annotation per question, unlike SQuAD which uses multiple; comparing directly without adjustment inflates SQuAD scores.
• MLQA contexts are on average 28% longer than SQuAD contexts, which can negatively impact model performance.
• Cross-lingual transfer from English often results in significant performance drops compared to monolingual English baselines, which is a key finding of the benchmark.

Evidence (verbatim from paper)

Table 5: F1 score and Exact Match on the MLQA test set for the cross-lingual transfer task (XLT)

Citation

@misc{lewis2019mlqa,
  title={MLQA: Evaluating Cross-lingual Extractive Question Answering},
  author={Lewis et al. (2019)},
  year={2019},
  note={arXiv:1910.07475}
}

• arXiv: 1910.07475