# Multilingual Sts Eval

> Evaluates the ability of sentence embedding models to capture semantic similarity across monolingual and cross-lingual sentence pairs. It probes how well vector spaces are aligned across different languages and whether fine-tuning on English NLI/STS data generalizes to other languages. Use when the user wants to benchmark on STS 2017, or asks about evaluating this task. Reports Spearman's rank correlation (ρ).

- Skill: `qhjqhj00/multilingual-sts-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/multilingual-sts-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/multilingual-sts-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/multilingual-sts-eval

---


# multilingual-sts-eval

> Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation — Reimers et al. (2020) (arXiv:2004.09813, 2020)

## What this evaluates

Evaluates the ability of sentence embedding models to capture semantic similarity across monolingual and cross-lingual sentence pairs. It probes how well vector spaces are aligned across different languages and whether fine-tuning on English NLI/STS data generalizes to other languages.

## Datasets

- **STS 2017** — total ?; splits: test (-1)

## Metrics

- `Spearman's rank correlation (ρ)` **(primary)** — range: [0, 100]
  - Computes cosine similarity between sentence embeddings, then calculates Spearman's rank correlation coefficient between the predicted similarities and the gold human-annotated scores. Reported as ρ × 100.

## Input / output format

**Input**: Pairs of sentences (monolingual or cross-lingual).

**Output**: Cosine similarity score between the two sentence embeddings.

## Scoring recipe

```python
scores = [cosine_similarity(embed(s1), embed(s2)) for s1, s2 in pairs]
rho = spearman_rank_correlation(scores, gold_scores)
return rho * 100
```

## Common pitfalls

- Using raw mBERT/XLM-R mean pooling without fine-tuning yields poor cross-lingual performance due to misaligned vector spaces.
- Confusing monolingual evaluation (Table 1) with cross-lingual evaluation (Table 2); performance drops significantly in cross-lingual settings for non-distilled models.

## Evidence (verbatim from paper)

> For the generate sentence embeddings we compute cosine similarity and, as recommended in (Reimers et al., 2016), compute the Spearman's rank correlation  $
ho$  between the computed score and the gold score.

## Citation

```bibtex
@misc{reimers2020multilingual,
  title={Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation},
  author={Reimers et al. (2020)},
  year={2020},
  note={arXiv:2004.09813}
}
```

- arXiv: 2004.09813