Triviaqa Eval

triviaqa-eval

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension — Mandar Joshi et al. (arXiv:1705.03551, 2017)

What this evaluates

This benchmark evaluates reading comprehension on complex, compositional trivia questions that require multi-sentence reasoning and handling high lexical variability. It tests a model's ability to locate and extract precise answers from large, noisy evidence documents across different domains.

Datasets

• TriviaQA — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• exact match (EM) (primary) — range: [0, 1]
  • Exact match (EM) and F1 over words in the answer(s). For questions that have Numerical and FreeForm answers, a single given answer is used as ground truth. For questions that have Wikipedia entities as answers, Wikipedia aliases are valid alongside the given answer.
• F1 — range: [0, 1]
  • Token-level F1 score computed between the predicted answer and the set of valid ground truth answers (including aliases).

Input / output format

Input: A natural language question paired with one or more evidence documents (from either the Wikipedia or Web domain).

Output: An extracted answer span or entity string.

Scoring recipe

def compute_metrics(predictions, gold_answers):
    em_scores = []
    f1_scores = []
    for pred, golds in zip(predictions, gold_answers):
        valid_answers = set(golds) | set(golds_aliases)
        em = 1.0 if pred in valid_answers else 0.0
        em_scores.append(em)
        pred_tokens = set(pred.split())
        gold_tokens = set()
        for g in valid_answers:
            gold_tokens |= set(g.split())
        if not pred_tokens or not gold_tokens:
            f1 = 0.0
        else:
            intersection = len(pred_tokens & gold_tokens)
            precision = intersection / len(pred_tokens)
            recall = intersection / len(gold_tokens)
            f1 = 2 * precision * recall / (precision + recall)
        f1_scores.append(f1)
    return sum(em_scores) / len(em_scores), sum(f1_scores) / len(f1_scores)

Common pitfalls

• Evaluation granularity differs by domain: question-level accuracy is reported for Wikipedia, while document-level accuracy is reported for the Web domain due to high information redundancy.
• Oracle score upper bounds are artificially capped below 100% because training documents are truncated to the first 400 tokens to manage training time.
• Entity answers require matching against a set of Wikipedia aliases, not just the exact ground truth string.

Evidence (verbatim from paper)

We use the same evaluation metrics as SQuAD – exact match (EM) and F1 over words in the answer(s). For questions that have Numerical and FreeForm answers, we use a single given answer as ground truth. For questions that have Wikipedia entities as answers, we use Wikipedia aliases as valid answer along with the given answer.

Citation

@misc{joshi2017triviaqa,
  title={TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension},
  author={Mandar Joshi et al.},
  year={2017},
  note={arXiv:1705.03551}
}

• arXiv: 1705.03551