Newsqa Eval

newsqa-eval

NewsQA: A Machine Comprehension Dataset — Trischler et al. (2016) (arXiv:1611.09830, 2016)

What this evaluates

Probes machine reading comprehension on real-world news articles by requiring models to extract answer spans from context based on natural-language questions. It specifically evaluates the ability to perform reasoning, synthesis, and contextual inference beyond simple keyword matching, while handling multiple valid phrasings for the same answer.

Datasets

• NewsQA — total 102841; splits: train (92549), val (5166), test (5126)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall computed over token-level overlap between the predicted answer span and the gold answer span.
• Exact Match (EM) — range: [0, 1]
  • Binary score of 1 if the predicted answer string exactly matches the gold answer string, 0 otherwise.
• BLEU — range: [0, 1]
  • Precision-based metric using a weighted average of n-gram matches between predicted and reference answers.
• CIDEr — range: [0, 1]
  • Metric that computes similarity using tf-idf weighted n-gram overlaps, designed to correlate better with human judgments.

Input / output format

Input: A news article (context) and a natural-language question.

Output: A text span extracted verbatim from the article that answers the question.

Scoring recipe

def compute_metrics(pred, gold):
    pred_tok = pred.split()
    gold_tok = gold.split()
    em = 1.0 if pred == gold else 0.0
    if not pred_tok or not gold_tok:
        f1 = 0.0
    else:
        common = Counter(pred_tok) & Counter(gold_tok)
        num_same = sum(common.values())
        prec = num_same / len(pred_tok)
        rec = num_same / len(gold_tok)
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return em, f1

Common pitfalls

• Strict exact match penalizes semantically correct answers that differ only in phrasing or formatting (e.g., '1996' vs 'in 1996').
• The reported experiments exclude unanswerable/null questions, focusing only on samples with crowdworker agreement.
• Simple n-gram metrics (BLEU/CIDEr) may not fully capture complex reasoning capabilities required by the dataset.

Evidence (verbatim from paper)

We used four performance measures: F1 and exact match (EM) scores (the same measures used by SQuAD), as well as BLEU and CIDEr... Performance of the baseline models and humans is measured by EM and F1 with the official evaluation script from SQuAD and listed in Table 4.

Citation

@misc{trischler2016newsqa,
  title={NewsQA: A Machine Comprehension Dataset},
  author={Trischler et al. (2016)},
  year={2016},
  note={arXiv:1611.09830}
}

• arXiv: 1611.09830