Qqp Paraphrase Generation Eval

qqp-paraphrase-generation-eval

Revisiting Paraphrase Question Generator using Pairwise Discriminator — Patro et al. (2019) (arXiv:1912.13149, 2019)

What this evaluates

Evaluates a model's ability to generate semantically equivalent paraphrase sentences from an input question, measuring lexical and semantic overlap with ground truth references. The benchmark probes sentence-level semantic understanding and generative fluency in a question-paraphrase setting.

Datasets

• Quora Question Pairs (QQP) — total 400000; splits: train (50000), train (100000), val (5000), test (30000); repo https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs

Metrics

• BLEU (primary) — range: [0, 1]
  • Standard n-gram precision metric with a brevity penalty to penalize overly short generations. Typically reported as BLEU-1 (unigram precision).
• METEOR — range: [0, 1]
  • Harmonic mean of unigram precision and recall, incorporating stemming and synonymy matching to better capture semantic similarity than exact n-gram overlap.
• TER — range: [0, 1]
  • Translation Error Rate; calculated as the number of edit operations (insertions, deletions, substitutions, shifts) required to match the reference, divided by the reference length.
• ROUGE — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation; measures n-gram overlap between the generated and reference text, focusing on recall.
• CIDEr — range: [0, 1]
  • Consensus-based Image Description Evaluation; uses TF-IDF weighting to measure how well the generated text matches the consensus of multiple reference texts.

Input / output format

Input: Original question text (string)

Output: Generated paraphrase question text (string)

Scoring recipe

def compute_metrics(predictions, references):
    bleu = nltk.translate.bleu_score.sentence_bleu(references, pred, weights=(1,0,0,0))
    meteor = nltk.translate.meteor_score.single_meteor_score(ref, pred)
    ter = ter_score(pred, ref)  # edit distance ops / ref length
    rouge = nltk.translate.rouge_score.rouge_l(ref, pred)['fmeasure']
    cider = nltk.translate.cider_score.cider_score(ref, pred)[0]
    return {'BLEU': bleu, 'METEOR': meteor, 'TER': ter, 'ROUGE': rouge, 'CIDEr': cider}

Common pitfalls

• The dataset split is non-standard: 50k/100k training pairs, 5k validation, and 30k test, following specific prior works rather than the official Quora split.
• Metrics are reported as absolute scores and percentage improvements over baselines, but exact tokenization/preprocessing details for BLEU/METEOR are not specified, which can cause score variance across implementations.

Evidence (verbatim from paper)

We observe that the score EDLPS method performs best across all scores (BLEU, ROUGE, METEOR, and CIDEr).

Citation

@misc{patro2019revisiting,
  title={Revisiting Paraphrase Question Generator using Pairwise Discriminator},
  author={Patro et al. (2019)},
  year={2019},
  note={arXiv:1912.13149}
}

• arXiv: 1912.13149