General Nlu Eval

general-nlu-eval

Does Knowledge Help General NLU? An Empirical Study — Xu et al. (2021) (arXiv:2109.00563, 2021)

What this evaluates

Evaluates whether integrating external knowledge (textual descriptions or embeddings) improves performance on general natural language understanding tasks compared to baseline pre-trained language models. It probes the model's ability to leverage external semantic information to enhance representation learning and decision-making across classification, regression, and sequence labeling benchmarks.

Datasets

• GLUE, Penn Treebank, CoNLL-2003 — total ?; splits: train (-1), val (-1); HF glue, ptb, conll2003

Metrics

• Accuracy / F1 / Pearson correlation / Matthew's correlation (primary) — range: [0, 1] | percent
  • Task-specific metrics: Accuracy for classification tasks, F1 for sequence labeling, Pearson correlation for STS-B, and Matthew's correlation for CoLA. Scores are averaged across tasks for overall comparison.

Input / output format

Input: Sentence or token sequence for classification/regression tasks; token sequence for sequence labeling tasks.

Output: Task-specific prediction: class label, regression value, or token-level sequence tags.

Scoring recipe

def compute_metric(preds, golds, task):
    if task in ['SST-2', 'MNLI', 'QQP', 'QNLI', 'MRPC', 'RTE']:
        return accuracy(golds, preds)
    elif task == 'CoLA':
        return matthews_corrcoef(golds, preds)
    elif task == 'STS-B':
        return pearson_corr(golds, preds)
    elif task in ['POS', 'NER']:
        return f1_score(golds, preds, average='micro')

Common pitfalls

• Reports medians over 5 random seeds, not means.
• Evaluation is performed on the development set rather than the official held-out test set.
• Task-specific metrics are used: Accuracy for classification, Matthew's correlation for CoLA, Pearson correlation for STS-B, and F1 for sequence labeling.

Evidence (verbatim from paper)

Metrics: Matt. corr., Acc., Pear. corr., F1. We report the median of results on the development set over five fixed random seeds for all tasks.

Citation

@misc{xu2021doesknowledgehelp,
  title={Does Knowledge Help General NLU? An Empirical Study},
  author={Xu et al. (2021)},
  year={2021},
  note={arXiv:2109.00563}
}

• arXiv: 2109.00563