Wep Verbalization Validity Eval

wep-verbalization-validity-eval

Probing neural language models for understanding of words of estimative probability — Sileo et al. (2022) (arXiv:2211.03358, 2022)

What this evaluates

Evaluates neural language models' ability to understand Words of Estimative Probability (WEP) by testing their capacity to distinguish valid from invalid probabilistic verbalizations and perform logical consistency checks in probabilistic reasoning.

Datasets

• WEP Reasoning 1 hop — total ?; splits: val (-1), test (-1)
• WEP Reasoning 2 hops — total ?; splits: val (-1), test (-1)
• WEP-UNLI — total ?; splits: val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of instances where the model correctly identifies the valid verbalization over the invalid one.

Input / output format

Input: Premise (context) and hypothesis (conclusion) pairs, or multiple-choice questions presenting a valid and an invalid WEP verbalization candidate.

Output: Binary classification label indicating whether the verbalization is valid or invalid, derived from logit scores combined via softmax.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return correct / len(gold_labels)

Common pitfalls

• Off-the-shelf causal and masked language models perform near chance without fine-tuning, as they lack explicit WEP semantics.
• Fine-tuning on MNLI does not transfer well to WEP reasoning tasks because MNLI contains very few WEP compositions.
• Likelihood scoring requires careful normalization (length-normalization or calibration) per dataset and model to be effective.

Evidence (verbatim from paper)

We conduct verbalization validity prediction (binary classification task of WEP correctness detection between two candidates) under two settings. We evaluate the rate at which valid verbalization is scored higher than invalid verbalization. We use a multiple-choice-question answering setup (we predict logit scores for the valid and invalid verbalization, combine their score with a softmax, then optimize the likelihood of the valid verbalization). The very low accuracy of causal and masked language models (first two rows) demonstrates how challenging the WEP-understanding tasks are.

Citation

@misc{sileo2022probing,
  title={Probing neural language models for understanding of words of estimative probability},
  author={Sileo et al. (2022)},
  year={2022},
  note={arXiv:2211.03358}
}

• arXiv: 2211.03358