Bucc Bitext Retrieval Eval

bucc-bitext-retrieval-eval

Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation — Reimers et al. (2020) (arXiv:2004.09813, 2020)

What this evaluates

Evaluates the capability of sentence embeddings to retrieve exact translation pairs (bitexts) from large monolingual corpora across different languages. It tests the model's ability to distinguish parallel sentences from semantically similar but non-parallel ones.

Datasets

• BUCC bitext mining task — total ?; splits: train (-1), test (-1)

Metrics

• F1 score (primary) — range: [0, 100]
  • Binary classification F1 score for identifying parallel sentence pairs above a learned threshold. Threshold is optimized on the training set.

Input / output format

Input: Sentence embeddings from two monolingual corpora (e.g., English and German).

Output: A similarity score computed via a margin function over cosine similarities and k-nearest neighbor averages, followed by a binary parallel/non-parallel decision.

Scoring recipe

def score(x, y, k=10):
    cos_xy = cosine_similarity(x, y)
    nn_x = k_nearest_neighbors(x, corpus_other_lang, k)
    nn_y = k_nearest_neighbors(y, corpus_src_lang, k)
    avg_nn = (sum(cosine_similarity(x, z) for z in nn_x) / (2*k)) + (sum(cosine_similarity(y, z) for z in nn_y) / (2*k))
    return cos_xy / avg_nn
threshold = optimize_threshold(train_scores, train_labels)
preds = [1 if score(x, y) > threshold else 0 for x, y in test_pairs]
return f1_score(test_labels, preds)

Common pitfalls

• BUCC contains false negatives: Wikipedia sentences labeled non-parallel are often actually valid translations, inflating false positives for good models.
• Models optimized for semantic similarity will score non-parallel but semantically similar sentences highly, which is penalized on BUCC but correct for semantic tasks.

Evidence (verbatim from paper)

Performance is measured using $F_{1}$ score. ... score(x, y) = margin(cos(x, y), sum_{z in NN_k(x)} cos(x,z)/(2k) + sum_{z in NN_k(y)} cos(y,z)/(2k)) with margin(a, b) = a/b.

Citation

@misc{reimers2020multilingual,
  title={Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation},
  author={Reimers et al. (2020)},
  year={2020},
  note={arXiv:2004.09813}
}

• arXiv: 2004.09813