Paws X Eval

paws-x-eval

PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification — Yang et al. (2019) (arXiv:1908.11828, 2019)

What this evaluates

PAWS-X evaluates a model's ability to identify paraphrases across multiple languages, specifically probing sensitivity to word order and syntactic structure under conditions of high lexical overlap. It measures how well models generalize cross-lingually when trained on machine-translated data versus zero-shot settings.

Datasets

• PAWS-X — total 23659; splits: train (-1), dev (-1), test (-1); repo https://github.com/google-research-datasets/paws

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified sentence pairs. Computed by comparing predicted labels against gold labels.
• AUC-PR — range: percent
  • Area under the precision-recall curve. For BERT, computed using probability scores for the positive class. For BOW and ESIM, computed using cosine similarity scores with a 0.5 threshold.

Input / output format

Input: A pair of sentences (a1, a2) in a single language (English, French, Spanish, German, Chinese, Japanese, or Korean).

Output: Binary classification label: 'match' (paraphrase) or 'not_MATCH' (non-paraphrase).

Scoring recipe

def compute_metrics(predictions, gold_labels, model_scores=None):
    # predictions, gold_labels: list of 'match'/'not_MATCH'
    accuracy = sum(1 for p, g in zip(predictions, gold_labels) if p == g) / len(gold_labels) * 100
    
    auc_pr = -1
    if model_scores is not None:
        gold_binary = [1 if g == 'match' else 0 for g in gold_labels]
        prec, rec, _ = precision_recall_curve(gold_binary, model_scores)
        auc_pr = auc(rec, prec)
    return accuracy, auc_pr

Common pitfalls

• Models may achieve high accuracy by exploiting translation artifacts or shared entity names rather than true cross-lingual structural understanding.
• Zero-shot evaluation on non-English languages is highly sensitive to the quality of the machine translation system used to generate training data, leading to performance gaps between Indo-European and CJK languages.
• Bag-of-words and simple similarity baselines fail because the dataset deliberately maximizes lexical overlap while varying word order, requiring explicit sensitivity to syntax.

Evidence (verbatim from paper)

We use two metrics: classification accuracy and area-under-curve scores of precision-recall curves (AUC-PR). For BERT, probability scores for the positive class is used to compute AUC-PR. For BOW and ESIM a cosine threshold of 0.5 is used to compute accuracy. In all experiments, the best model checkpoint is chosen based on accuracy on development sets and report results on testing sets.

Citation

@misc{yang2019pawsx,
  title={PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification},
  author={Yang et al. (2019)},
  year={2019},
  note={arXiv:1908.11828}
}

• arXiv: 1908.11828