Mnli Anli Eval

mnli-anli-eval

Does Putting a Linguist in the Loop Improve NLU Data Collection? — Parrish et al. (2021) (arXiv:2104.07179, 2021)

What this evaluates

Evaluates the out-of-domain generalization and robustness of NLI models trained under different data collection protocols. It measures how well models perform on held-out, genre-diverse, and adversarial benchmarks compared to in-domain validation performance.

Datasets

• MNLI-mismatched — total ?; splits: test (-1); HF mnli
• ANLI — total ?; splits: test (-1); HF anli

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of examples in the test set. Reported as average accuracy across 10 bootstrap runs (90% sampling) with confidence intervals.

Input / output format

Input: Premise-hypothesis pairs with a natural language inference label (entailment, contradiction, or neutral).

Output: Predicted label for each premise-hypothesis pair.

Scoring recipe

accuracies = []
for _ in range(10):
    sample = bootstrap_sample(gold, size=0.9)
    preds = model.predict(sample)
    acc = sum(p == g for p, g in zip(preds, sample.labels)) / len(sample)
    accuracies.append(acc)
return mean(accuracies), confidence_interval(accuracies)

Common pitfalls

• Evaluating on MNLI-matched instead of MNLI-mismatched, which would artificially inflate performance due to genre overlap with the training data.
• Reporting point estimates without the required bootstrapping procedure (10 runs with 90% sampling) to compute confidence intervals.
• Using the full 3.5k examples per round for training instead of the specified 3k training / 500 validation split.

Evidence (verbatim from paper)

After the final round of data collection, we evaluate models trained on our data on MNLI-mismatched (Williams et al., 2018) and ANLI (Nie et al., 2020). The MNLI corpus includes two evaluation sets, MNLI-matched and MNLI-mismatched, with examples sourced from different genres. We evaluate on MNLI-mismatched, as we source our premise sentences from an MNLI-matched genre. Evaluating on held-out sets allows us to test if our interventions lead to increased model accuracy on datasets generated through different protocols or from different sources while ensuring that we do not overly tune our feedback to these benchmarks. We estimate average accuracy and confidence intervals by fine-tuning 10 additional models with a sample of 90% of the collected training data.

Citation

@misc{parrish2021linguist,
  title={Does Putting a Linguist in the Loop Improve NLU Data Collection?},
  author={Parrish et al. (2021)},
  year={2021},
  note={arXiv:2104.07179}
}

• arXiv: 2104.07179