Cola Eval

cola-eval

Neural Network Acceptability Judgments — Warstadt et al. (2018) (arXiv:1805.12471, 2018)

What this evaluates

This benchmark evaluates a model's ability to classify English sentences as grammatically acceptable or unacceptable. It probes syntactic competence by measuring performance on both in-domain and out-of-domain linguistic data.

Datasets

• CoLA — total 10657; splits: train (-1), dev (-1), test (-1)

Metrics

• MCC (primary) — range: [-1, 1]
  • Matthews Correlation Coefficient, measuring the quality of binary classifications. Calculated as (TPTN - FPFN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).
• Accuracy — range: [0, 1]
  • The proportion of correctly classified sentences out of the total number of sentences.

Input / output format

Input: A single English sentence.

Output: A binary label: 1 for acceptable, 0 for unacceptable.

Scoring recipe

def score(predictions, gold):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    tn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 0)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    denom = ((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)) ** 0.5
    mcc = (tp * tn - fp * fn) / denom if denom > 0 else 0.0
    return {'accuracy': acc, 'mcc': mcc}

Common pitfalls

• Threshold tuning for unsupervised baselines (e.g., Lau et al.'s WLPM) is performed via 10-fold cross-validation on the CoLA test set itself, which can inflate performance if not strictly separated from training.
• Human performance is reported as both 'Average' and 'Aggregate' agreement, which differ significantly and should not be conflated with model scores.
• The out-of-domain evaluation set is not described in this section, making replication of that specific metric difficult.

Evidence (verbatim from paper)

We evaluate these classifiers on CoLA without CoLA training. ... We train 20 pooling classifiers end-to-end on real/fake data with BNC embeddings, 20 with GloVe, and 20 with ELMo-style embeddings for up to 7 days or until completing 4 epochs without improving in development MCC.

Citation

@misc{warstadt2018neural,
  title={Neural Network Acceptability Judgments},
  author={Warstadt et al. (2018)},
  year={2018},
  note={arXiv:1805.12471}
}

• arXiv: 1805.12471