Rumteb Eval

rumteb-eval

The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design — Snegirev et al. (2024) (arXiv:2408.12503, 2024)

What this evaluates

Evaluates Russian text embedding models across semantic similarity, classification, retrieval, and reranking tasks to measure their effectiveness in understanding and retrieving Russian language content.

Datasets

• ruMTEB — total ?; splits: test (-1); repo https://github.com/embeddings-benchmark/mteb

Metrics

• cosine_spearman (primary) — range: [-1, 1]
  • Spearman rank correlation between gold semantic similarity scores and the cosine similarity of predicted embedding vectors. Standard for STS tasks in MTEB.
• accuracy — range: [0, 1]
  • Proportion of correctly classified instances using a linear probe or nearest-neighbor classifier on the predicted embeddings.
• map — range: [0, 1]
  • Mean Average Precision across queries, measuring the quality of ranked retrieval results against gold relevant documents.
• ndcg@k — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff k, evaluating the ranking quality of retrieved or reranked documents against relevance labels.

Input / output format

Input: Text pairs (for semantic similarity), single texts (for classification), or query-document pairs (for retrieval and reranking).

Output: Fixed-dimensional embedding vectors for each input text.

Scoring recipe

def evaluate_rumteb(predictions, gold, task_type):
    if task_type == 'STS':
        return spearman_corr(gold, cosine_sim(predictions))
    elif task_type == 'Classification':
        return accuracy(classify(predictions, gold))
    elif task_type in ['Retrieval', 'Reranking']:
        return map_at_k(gold, predictions)
    # MTEB framework handles task-specific metric aggregation automatically

Common pitfalls

• Using a different MTEB framework version than v1.14.12 specified in the paper, which can alter task definitions and metric calculations.
• Failing to distinguish between the 17 newly constructed tasks and the 6 adapted multilingual tasks, which may have different evaluation splits or label distributions.
• Not pinning the exact task definitions from the MTEB PRs (#815, #881) referenced in the text, leading to inconsistencies with the reported baseline results.

Evidence (verbatim from paper)

We evaluate all models in the same environments and scenarios by the procedure described in[3.1]. We use MTEB framework171717https://github.com/embeddings-benchmark/mteb/tree/1.14.12 for evaluation where we integrated evaluation on the new ruMTEB tasks181818https://github.com/embeddings-benchmark/mteb/pull/815191919https://github.com/embeddings-benchmark/mteb/pull/881

Citation

@misc{snegirev2024rumteb,
  title={The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design},
  author={Snegirev et al. (2024)},
  year={2024},
  note={arXiv:2408.12503}
}

• arXiv: 2408.12503