Megawika Eval

megawika-eval

MegaWika: Millions of reports and their sources across 50 diverse languages — Barham et al. (2023) (arXiv:2307.07049, 2023)

What this evaluates

Evaluates the quality and evidential support of automatically generated Wikipedia question-answer pairs and their cited source documents. It probes whether sources actually contain the information claimed in passages, and measures the strength of support for QA pairs.

Datasets

• MegaWika — total ?; splits: val (1500), test (1000)

Metrics

• answerability (primary) — range: [0, 3]
  • Human annotators rate the question-answer pair against the source document on a 3-point scale: 1 (completely unanswerable), 2 (partially answerable), 3 (indisputably answerable and correct). Instances failing sanity checks are assigned a strength of 0.

Input / output format

Input: Question, answer span, and cited source document text. For source support evaluation, a highlighted passage span evoking a FrameNet frame is also provided.

Output: Scalar rating (0–3) for answerability/evidential support strength, or binary majority vote for event containment.

Scoring recipe

def score_answerability(predictions, gold):
    # predictions: list of lists, each inner list contains ratings from redundant annotators
    # gold: not used in this manual protocol
    final_scores = []
    for pred_list in predictions:
        # Aggregate redundant annotator ratings via average
        score = sum(pred_list) / len(pred_list)
        final_scores.append(score)
    return sum(final_scores) / len(final_scores)

Common pitfalls

• Assuming cited Wikipedia sources automatically contain the claimed information; only ~48% actually support the highlighted events.
• Confusing answerability given the Wikipedia passage versus answerability given the cited source document, which yield significantly different scores (2.38/3 vs 1.67/3).
• Including citation-like or table-like text in the evaluation set without filtering, which degrades quality.

Evidence (verbatim from paper)

We assessed: (1) the quality of the passage extraction, (2) the quality of the source document scrape, (3) the fluency of the generated question, (4) the reasonableness of the question, (5) the answerability of the question given the Wiki passage, (6) the answerability of the question given the source document, and (7) the correctness of the selected answer span.

Citation

@misc{barham2023megawika,
  title={MegaWika: Millions of reports and their sources across 50 diverse languages},
  author={Barham et al. (2023)},
  year={2023},
  note={arXiv:2307.07049}
}

• arXiv: 2307.07049