Tatoeba Similarity Search Eval

tatoeba-similarity-search-eval

Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation — Reimers et al. (2020) (arXiv:2004.09813, 2020)

What this evaluates

Evaluates cross-lingual sentence retrieval accuracy for low-resource languages by finding the most similar sentence in a target language corpus for each source sentence. It probes the alignment quality of vector spaces for languages with limited parallel data.

Datasets

• Tatoeba test setup (LASER) — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 100]
  • Fraction of correctly retrieved parallel sentences. Computed for both directions (English to target language and target language to English) and averaged.

Input / output format

Input: Sentence embeddings for English and target language sentences.

Output: Index of the most similar sentence in the target language corpus (or vice versa) based on cosine similarity.

Scoring recipe

correct = 0
total = 0
for direction in ['en->xx', 'xx->en']:
    for src_sent, gold_tgt_sent in pairs:
        src_emb = embed(src_sent)
        tgt_embs = [embed(t) for t in tgt_corpus]
        pred_idx = argmax(cosine_similarity(src_emb, tgt_embs))
        if tgt_corpus[pred_idx] == gold_tgt_sent:
            correct += 1
        total += 1
return (correct / total) * 100

Common pitfalls

• Low-resource languages like Tagalog and Tatar were not in XLM-R's pre-training vocabulary, yet distillation still yields high accuracy, showing robustness to OOV vocabulary.
• Evaluation only covers 4 specific low-resource languages (KA, SW, TL, TT) and may not generalize to all low-resource settings.

Evidence (verbatim from paper)

Evaluation is done by finding for all sentences the most similar sentence in the other language using cosine similarity. Accuracy is computed for both directions (English to the other language and back).

Citation

@misc{reimers2020multilingual,
  title={Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation},
  author={Reimers et al. (2020)},
  year={2020},
  note={arXiv:2004.09813}
}

• arXiv: 2004.09813