Pl Mteb Eval

pl-mteb-eval

PL-MTEB: Polish Massive Text Embedding Benchmark — Poświata et al. (2024) (arXiv:2405.10138, 2024)

What this evaluates

Evaluates Polish and multilingual text embedding models across 28 tasks spanning classification, clustering, pair classification, retrieval, and semantic textual similarity. It measures how well embeddings capture semantic relationships, support downstream classification, cluster documents, and retrieve relevant documents in Polish.

Datasets

• PL-MTEB — total ?; splits: test (-1); repo https://github.com/rafalposwiata/pl-mteb

Metrics

• accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.
• v-measure — range: [0, 1]
  • Harmonic mean of homogeneity and completeness for clustering quality, measuring how well clusters match ground-truth labels.
• average precision score based on cosine similarity — range: [0, 1]
  • Computes cosine similarity between all embedding pairs, then calculates the average precision score against binary relevance labels.
• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring the quality of the ranked retrieval list against ground-truth relevant documents.
• Spearman correlation based on cosine similarity — range: [-1, 1]
  • Spearman rank correlation coefficient computed between cosine similarity scores of embedding pairs and their human-assigned similarity scores.

Input / output format

Input: Single texts or text pairs provided as strings. Models generate fixed-dimensional dense vectors (embeddings) for each input.

Output: Dense embedding vectors per input text. For retrieval tasks, a ranked list of candidate documents per query.

Scoring recipe

def compute_metrics(predictions, gold, task_type):
    if task_type == 'classification':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'clustering':
        return v_measure_score(gold, predictions)
    elif task_type == 'pair_classification':
        sims = cosine_similarity(predictions)
        return average_precision(sims, gold)
    elif task_type == 'retrieval':
        sims = cosine_similarity(predictions)
        return ndcg_at_k(sims, gold, k=10)
    elif task_type == 'sts':
        sims = cosine_similarity(predictions)
        return spearmanr(sims, gold).correlation
    return mean([accuracy, v_measure, ap, ndcg, spearman])

Common pitfalls

• MSMARCO-PL uses the dev split instead of the test split, unlike all other datasets in the benchmark.
• Pair classification and STS metrics rely on cosine similarity between embeddings rather than direct label prediction or regression.
• The overall benchmark score is the arithmetic mean of the task-type averages across all 28 tasks, not a simple average of all individual task scores.

Evidence (verbatim from paper)

Table 2: Evaluation results on classification tasks using accuracy metric. The best score for a given column is marked in bold, and the second best is underlined. ... Table 5: Evaluation results on retrieval tasks using nDCG@10. ... Table 6: Evaluation results on STS tasks using Spearman correlation based on cosine similarity.

Citation

@misc{poswiata2024plmteb,
  title={PL-MTEB: Polish Massive Text Embedding Benchmark},
  author={Poświata et al. (2024)},
  year={2024},
  note={arXiv:2405.10138}
}

• arXiv: 2405.10138