Semantic Textual Similarity Eval

semantic-textual-similarity-eval

SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation — Cer et al. (2017) (arXiv:1708.00055, 2017)

What this evaluates

Evaluates a model's ability to quantify the degree of semantic similarity between pairs of sentences, including multilingual and cross-lingual contexts. It probes fine-grained semantic matching and cross-lingual generalization rather than binary paraphrase detection.

Datasets

• SemEval-2017 STS — total ?; splits: test (-1)

Metrics

• Pearson correlation (primary) — range: [0, 1]
  • Pearson correlation coefficient computed between the model's predicted similarity scores and the human-annotated reference scores for each sentence pair.

Input / output format

Input: A pair of sentences (monolingual or cross-lingual).

Output: A real-valued similarity score, typically on a 0 to 5 ordinal scale.

Scoring recipe

import numpy as np
from scipy.stats import pearsonr

def evaluate(predictions, gold):
    # predictions, gold: list/array of floats (0-5)
    corr, _ = pearsonr(gold, predictions)
    return corr

Common pitfalls

• Using Spearman correlation instead of the specified Pearson correlation, which can yield different results on this scale.
• Treating the task as binary paraphrase detection rather than fine-grained similarity scoring.
• Failing to align cross-lingual sentence pairs correctly before scoring, leading to artificially low correlations.

Evidence (verbatim from paper)

STS is the assessment of pairs of sentences according to their degree of semantic similarity. The task involves producing real-valued similarity scores for sentence pairs. Performance is measured by the Pearson correlation of machine scores with human judgments. The ordinal scale in Table 1 guides human annotation, ranging from 0 for no meaning overlap to 5 for meaning equivalence.

Citation

@misc{cer2017semeval,
  title={SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation},
  author={Cer et al. (2017)},
  year={2017},
  note={arXiv:1708.00055}
}

• arXiv: 1708.00055