Paws Eval

paws-eval

PAWS: Paraphrase Adversaries from Word Scrambling — Yuan Zhang et al. (2019) (arXiv:1904.01130, 2019)

What this evaluates

This benchmark evaluates a model's ability to identify paraphrases in sentence pairs that share high lexical overlap but differ in meaning due to word order and syntactic structure. It specifically probes sensitivity to non-local contextual information and adversarial word scrambling, revealing whether models rely on superficial word matching rather than true semantic understanding.

Datasets

• PAWS_QQP — total ?; splits: dev (-1)
• PAWS_Wiki — total ?; splits: test (-1)

Metrics

• classification accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified sentence pairs out of the total. A fixed probability threshold of 0.5 is applied to model outputs before computing the ratio.
• AUC — range: [0, 1]
  • Area under the precision-recall curve, computed across all classification thresholds to measure the trade-off between precision and recall.

Input / output format

Input: Pairs of sentences (e.g., two questions or two Wikipedia sentences) to be classified as paraphrases or non-paraphrases.

Output: Binary classification label (paraphrase vs. non-paraphrase) or a continuous probability score used for thresholding and AUC computation.

Scoring recipe

# Accuracy
preds = [1 if p >= 0.5 else 0 for p in probs]
acc = sum(p == g for p, g in zip(preds, gold)) / len(gold)

# AUC (Precision-Recall)
prec, rec, _ = precision_recall_curve(gold, probs)
auc = np.trapz(prec, rec)

Common pitfalls

• Models frequently drop to near-chance accuracy (<40%) because they over-rely on superficial lexical overlap and ignore word order/syntax.
• PAWS_QQP evaluation is strictly on the development set, as the authors explicitly note it lacks a test set.
• AUC is computed on precision-recall curves rather than ROC curves, which is critical for correctly assessing performance on this specific task.

Evidence (verbatim from paper)

We use two metrics: classification accuracy and area-under-curve (AUC) scores of precision-recall curves. For all classification models, 0.5 is the threshold used to compute accuracy. We report results on testing sets for QQP and PAWS_Wiki, and on the development set for PAWS_QQP (which has no test set).

Citation

@misc{zhang2019paws,
  title={PAWS: Paraphrase Adversaries from Word Scrambling},
  author={Yuan Zhang et al. (2019)},
  year={2019},
  note={arXiv:1904.01130}
}

• arXiv: 1904.01130