Boolq Eval

boolq-eval

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions — Clark et al. (2019) (arXiv:1905.10044, 2019)

What this evaluates

This benchmark evaluates a model's ability to answer naturally occurring yes/no questions based on a provided passage. It probes complex inferential reasoning and non-factoid inference, requiring the model to go beyond simple keyword matching or shallow statistical features to determine entailment or contradiction between the question and the passage.

Datasets

• BoolQ — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • The proportion of correctly predicted yes/no answers out of the total number of instances in the evaluation split. Calculated as the number of correct predictions divided by the total number of predictions.

Input / output format

Input: A natural language yes/no question paired with a multi-sentence passage (context) from which the answer must be inferred.

Output: A binary classification label: 'yes' or 'no'.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models can achieve ~62% accuracy by simply predicting 'yes' for every question due to class imbalance, so the majority-class baseline must be reported and compared against.
• Shallow features like word overlap or TF-IDF are insufficient and often perform worse than the majority baseline, indicating the task requires deep semantic reasoning rather than lexical matching.

Evidence (verbatim from paper)

In general, we found features such as word overlap or TF-IDF statistics were not sufficient to achieve better than the majority-class baseline accuracy (62.17% on the dev set).

Citation

@misc{clark2019boolq,
  title={BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions},
  author={Clark et al. (2019)},
  year={2019},
  note={arXiv:1905.10044}
}

• arXiv: 1905.10044