qhjqhj00 vn-mteb-eval
VN-MTEB: Vietnamese Massive Text Embedding Benchmark — Loc Pham et al. (2025) (arXiv:2507.21500, 2025)
What this evaluates
Evaluates the quality of Vietnamese text embeddings across six standard information retrieval and NLP tasks. It probes a model's ability to capture semantic similarity, perform document retrieval, classify text, cluster documents, and rank relevant passages in Vietnamese.
Datasets
- VN-MTEB — total 41; splits: test (-1)
Metrics
Average Task Score(primary) — range: percent- Mean of the primary evaluation metric (e.g., NDCG@10 for retrieval, accuracy for classification, cosine similarity for STS) computed across all 41 datasets spanning six task categories. Scores are reported as percentages.
Input / output format
Input: Vietnamese text instances formatted per task: (query, corpus) for retrieval/reranking, (sentence1, sentence2) for STS/pair classification, single text for classification/clustering.
Output: Embedding vectors for each text instance. For retrieval/reranking, cosine similarity scores between query and document embeddings are computed to produce ranked lists. For classification/clustering/pair classification, predicted labels or cluster assignments are generated.
Scoring recipe
def compute_vn_mteb_score(predictions, gold, task_type):
if task_type == 'retrieval':
scores = cosine_similarity(predictions, gold)
return ndcg_at_k(scores, k=10)
elif task_type == 'sts':
return pearson_correlation(predictions, gold)
elif task_type in ['classification', 'pair_classification']:
return accuracy(predictions, gold)
elif task_type == 'clustering':
return normalized_mutual_information(predictions, gold)
elif task_type == 'reranking':
scores = cosine_similarity(predictions, gold)
return map_at_k(scores, k=10)
return 0.0
avg_score = mean([compute_vn_mteb_score(preds, golds[t], t) for t in tasks])
return avg_score * 100
Common pitfalls
- Using absolute positional embeddings (APE) instead of rotary embeddings (RoPE) significantly degrades performance on Vietnamese text.
- Failing to filter out low-quality translated samples can introduce mixed-language noise; the benchmark uses a cosine similarity threshold of 0.8 to discard semantically poor translations.
- Applying a single metric across all tasks is incorrect; each of the six tasks (retrieval, classification, clustering, pair classification, reranking, STS) requires its own specific evaluation metric.
Evidence (verbatim from paper)
The column "Avg." represents the mean of the average scores across all tasks. In the model evaluation process, we adhere to the methodology outlined in the MTEB task Muennighoff et al. ([2023]). Specifically, we employ the model to embed both the queries and the corpus documents for the Retrieval task. Cosine similarity is then used to compute the similarity scores between each query and document. Next, we rank the corpus documents for each query based on their respective similarity scores and calculate various evaluation metrics.
Citation
@misc{pham2025vn_mteb,
title={VN-MTEB: Vietnamese Massive Text Embedding Benchmark},
author={Loc Pham et al. (2025)},
year={2025},
note={arXiv:2507.21500}
}
- arXiv: 2507.21500