Blimp Eval

blimp-eval

BLiMP: The Benchmark of Linguistic Minimal Pairs for English — Warstadt et al. (2019) (arXiv:1912.00582, 2019)

What this evaluates

This benchmark probes language models' sensitivity to grammatical acceptability contrasts across 12 linguistic phenomena. It evaluates whether models can reliably distinguish acceptable sentences from minimally ungrammatical ones, revealing strengths in morphological agreement and weaknesses in complex syntactic and semantic constraints.

Datasets

• BLiMP — total 67000; splits: test (67000)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of minimal pairs in which the model assigns a higher probability to the acceptable sentence than to the ungrammatical one. Computed overall across 67,000 pairs and reported per linguistic phenomenon.

Input / output format

Input: Pairs of sentences (one acceptable, one ungrammatical) for each linguistic paradigm.

Output: Predicted log-probabilities or probabilities for each sentence in the pair.

Scoring recipe

correct = 0
for pair in dataset:
    p_acceptable = model.log_prob(pair.acceptable)
    p_ungrammatical = model.log_prob(pair.ungrammatical)
    if p_acceptable > p_ungrammatical:
        correct += 1
return correct / len(dataset)

Common pitfalls

• Interpreting positive results on a general phenomenon as proof of human-like linguistic knowledge without testing generalization to nonce words or out-of-distribution examples.
• Assuming the forced-choice minimal pair task fully captures syntactic island knowledge, as it cannot disentangle gap-filling from island constraints without factorial designs.
• Attributing performance differences primarily to model architecture when training data scale is the main driver of accuracy variations.

Evidence (verbatim from paper)

An LM's overall accuracy on BLiMP is simply the proportion of the 67,000 minimal pairs in which the model assigns a higher probability to the acceptable sentence.

Citation

@misc{warstadt2019blimp,
  title={BLiMP: The Benchmark of Linguistic Minimal Pairs for English},
  author={Warstadt et al. (2019)},
  year={2019},
  note={arXiv:1912.00582}
}

• arXiv: 1912.00582