Orb Eval

orb-eval

ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension — Dua et al. (2019) (arXiv:1912.12598, 2019)

What this evaluates

Evaluates machine reading comprehension models across diverse linguistic phenomena such as coreference resolution, temporal logic, and causal inference. It tests generalization capabilities by applying synthetic out-of-distribution augmentations to questions, ensuring models rely on reading comprehension rather than external information retrieval.

Datasets

• ORB Benchmark (NewsQA, Quoref, DROP, SQuAD 1.1, SQuAD 2.0, ROPES, DuoRC, NarrativeQA) — total ?; splits: dev (-1), test (-1)

Metrics

• EM (primary) — range: [0, 1]
  • 1.0 if the predicted answer exactly matches the gold answer (case-insensitive), 0.0 otherwise. Averaged across all instances.
• Token F1 — range: [0, 1]
  • Harmonic mean of token-level precision and recall between predicted and gold answers. Precision = common tokens / predicted tokens. Recall = common tokens / gold tokens. Averaged across instances.

Input / output format

Input: A context passage and a natural language question.

Output: Predicted answer span (start and end token indices in the context) or free-form text. For free-form datasets, training uses the context span with the highest ROUGE-L to the gold answer, but evaluation uses the original gold answer.

Scoring recipe

from collections import Counter
def compute_em_f1(preds, golds):
    em_scores = []
    f1_scores = []
    for pred, gold in zip(preds, golds):
        pred_tokens = pred.lower().split()
        gold_tokens = gold.lower().split()
        em_scores.append(1.0 if pred_tokens == gold_tokens else 0.0)
        if len(gold_tokens) == 0:
            f1_scores.append(0.0)
            continue
        common = Counter(pred_tokens) & Counter(gold_tokens)
        num_same = sum(common.values())
        precision = num_same / len(pred_tokens) if len(pred_tokens) > 0 else 0.0
        recall = num_same / len(gold_tokens)
        f1 = (2 * precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
        f1_scores.append(f1)
    return sum(em_scores) / len(em_scores), sum(f1_scores) / len(f1_scores)

Common pitfalls

• Catastrophic forgetting occurs when sampling randomly from pooled datasets; requires uniform per-dataset sampling initially.
• Free-form answers in DuoRC/NarrativeQA are converted to ROUGE-L best spans for training labels, but evaluation uses original gold answers.
• Performance is heavily dominated by SQuAD 1.1, masking weaknesses on other datasets.

Evidence (verbatim from paper)

This helped improve the performance on several dataset by 3 - 4% in EM, however, there is still a lot of room for improvement on this front.

Citation

@misc{dua2019orb,
  title={ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension},
  author={Dua et al. (2019)},
  year={2019},
  note={arXiv:1912.12598}
}

• arXiv: 1912.12598