Multilingual Sts Eval

multilingual-sts-eval

Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation — Reimers et al. (2020) (arXiv:2004.09813, 2020)

What this evaluates

Evaluates the ability of sentence embedding models to capture semantic similarity across monolingual and cross-lingual sentence pairs. It probes how well vector spaces are aligned across different languages and whether fine-tuning on English NLI/STS data generalizes to other languages.

Datasets

• STS 2017 — total ?; splits: test (-1)

Metrics

• Spearman's rank correlation (ρ) (primary) — range: [0, 100]
  • Computes cosine similarity between sentence embeddings, then calculates Spearman's rank correlation coefficient between the predicted similarities and the gold human-annotated scores. Reported as ρ × 100.

Input / output format

Input: Pairs of sentences (monolingual or cross-lingual).

Output: Cosine similarity score between the two sentence embeddings.

Scoring recipe

scores = [cosine_similarity(embed(s1), embed(s2)) for s1, s2 in pairs]
rho = spearman_rank_correlation(scores, gold_scores)
return rho * 100

Common pitfalls

• Using raw mBERT/XLM-R mean pooling without fine-tuning yields poor cross-lingual performance due to misaligned vector spaces.
• Confusing monolingual evaluation (Table 1) with cross-lingual evaluation (Table 2); performance drops significantly in cross-lingual settings for non-distilled models.

Evidence (verbatim from paper)

For the generate sentence embeddings we compute cosine similarity and, as recommended in (Reimers et al., 2016), compute the Spearman's rank correlation $ ho$ between the computed score and the gold score.

Citation

@misc{reimers2020multilingual,
  title={Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation},
  author={Reimers et al. (2020)},
  year={2020},
  note={arXiv:2004.09813}
}

• arXiv: 2004.09813