Senteval Eval

senteval-eval

SentEval: An Evaluation Toolkit for Universal Sentence Representations — Conneau et al. (2018) (arXiv:1803.05449, 2018)

What this evaluates

Evaluates the transferability and quality of universal sentence embeddings across a standardized suite of downstream tasks. It probes capabilities in sentiment classification, natural language inference, semantic textual similarity, and cross-modal image-caption retrieval using fixed hyperparameters and consistent preprocessing.

Datasets

• MR — total 11000; splits: train (-1)
• CR — total 4000; splits: train (-1)
• SUBJ — total 10000; splits: train (-1)
• MPQA — total 11000; splits: train (-1)
• TREC — total 6000; splits: train (-1)
• SST-2 — total 70000; splits: train (-1)
• SST-5 — total 12000; splits: train (-1)
• SNLI — total 560000; splits: train (-1)
• SICK-E — total 10000; splits: train (-1)
• SICK-R — total 10000; splits: train (-1)
• STS14 — total 4500; splits: train (-1)
• MRPC — total 5700; splits: train (-1)
• COCO — total 565000; splits: train (113000), val (5000), test (5000)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels over the total number of instances. Used for binary/multi-class classification, NLI, and paraphrase detection.
• pearson (primary) — range: [-1, 1]
  • Pearson product-moment correlation coefficient between human-annotated similarity scores and model-predicted scores (or cosine similarity for unsupervised STS).
• spearman — range: [-1, 1]
  • Spearman rank-order correlation coefficient between human-annotated similarity scores and model-predicted scores. Reported alongside Pearson for STS tasks.
• recall@K — range: [0, 1]
  • Percentage of queries where the true matching image/caption appears in the top K retrieved results. Evaluated for K in {1, 5, 10}.
• median_rank — range: other
  • Median of the ranks of the true matching image/caption across all queries. Lower values indicate better retrieval performance.

Input / output format

Input: Single sentences for classification and STS tasks; sentence pairs for NLI, STS, and paraphrase detection; image-caption pairs for retrieval.

Output: Class labels (e.g., pos/neg, entailment/neutral/contradiction), continuous similarity scores in [0, 5], or ranked lists of images/captions.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type in ['classification', 'nli', 'paraphrase']:
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task_type == 'sts':
        from scipy.stats import pearsonr, spearmanr
        p_corr, _ = pearsonr(golds, predictions)
        s_corr, _ = spearmanr(golds, predictions)
        return p_corr, s_corr
    elif task_type == 'retrieval':
        ranks = [rank_of_true_match(q) for q in queries]
        recall_k = [sum(1 for r in ranks if r <= k) / len(ranks) for k in [1, 5, 10]]
        med_r = median(ranks)
        return recall_k, med_r

Common pitfalls

• Using inconsistent preprocessing or hyperparameters across models, which breaks fair comparison (SentEval enforces a fixed pipeline).
• Confusing supervised STS tasks (SICK-R, STS14) that require training a predictor, with unsupervised STS tasks (STS12-16) that only compute cosine similarity between fixed embeddings.
• Reporting only Pearson correlation for STS; the protocol requires both Pearson and Spearman, plus their average/weighted average across subtasks.

Evidence (verbatim from paper)

For semantic relatedness, which consists of predicting a semantic score between 0 and 5 from two input sentences, we follow the approach of Tai et al. (2015a) and learn to predict the probability distribution of relatedness scores. SentEval reports Pearson and Spearman correlation.

Citation

@misc{conneau2018senteval,
  title={SentEval: An Evaluation Toolkit for Universal Sentence Representations},
  author={Conneau et al. (2018)},
  year={2018},
  note={arXiv:1803.05449}
}

• arXiv: 1803.05449