Mrqa 2019 Shared Task Eval

mrqa-2019-shared-task-eval

MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension — Fisch et al. (2019) (arXiv:1910.09753, 2019)

What this evaluates

Evaluates out-of-domain generalization in extractive reading comprehension by testing models on held-out datasets from diverse domains (crowdsourced, synthetic, domain experts, Wikipedia, education, etc.) that were not seen during training.

Datasets

• MRQA 2019 Shared Task — total ?; splits: Split I (-1), Split II (-1), Split III (-1); repo https://github.com/mrqa/MRQA-Shared-Task-2019

Metrics

• F1 (primary) — range: [0, 1]
  • Standard extractive QA F1 score, computed as the harmonic mean of exact-match precision and recall over predicted answer spans versus gold answer spans in the context.

Input / output format

Input: A context passage and a natural language question.

Output: A text span extracted from the context, corresponding to the start and end token indices of the answer.

Scoring recipe

def compute_macro_f1(predictions, golds):
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        pred_tokens = set(pred.split())
        gold_tokens = set(gold.split())
        if not gold_tokens:
            f1_scores.append(0.0)
            continue
        overlap = len(pred_tokens & gold_tokens)
        prec = overlap / len(pred_tokens) if pred_tokens else 0.0
        rec = overlap / len(gold_tokens)
        f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Models are ranked on the macro-averaged F1 across 12 held-out test datasets (Split II + III), not on in-domain performance (Split I).
• The task strictly requires extractive spans; models generating abstractive answers are penalized or disqualified.
• Data sampling restrictions: participants were only allowed to use the provided training data from six datasets, disallowing external QA data.

Evidence (verbatim from paper)

The teams are ranked by the F1 scores on the hidden testing portions of the 12 datasets (Split II and III in Section 3.1). Table 3 lists the macro-averaged F1 scores of all the submissions on both the development and testing portions of the MRQA dataset.

Citation

@misc{fisch2019mrqa,
  title={MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension},
  author={Fisch et al. (2019)},
  year={2019},
  note={arXiv:1910.09753}
}

• arXiv: 1910.09753