Famteb Eval

famteb-eval

FaMTEB: Massive Text Embedding Benchmark in Persian Language — Zinvandi et al. (2025) (arXiv:2502.11571, 2025)

What this evaluates

Evaluates the effectiveness of text embedding models across seven diverse tasks (classification, clustering, pair classification, reranking, retrieval, semantic textual similarity, and summary retrieval) specifically for the Persian language.

Datasets

• FaMTEB — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified or matched samples out of the total evaluated samples.
• ndcg@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring ranking quality for retrieval tasks.
• correlation_coefficient — range: [-1, 1]
  • Correlation between model-generated labels or similarity scores and human annotator labels for ordinal or continuous tasks.

Input / output format

Input: Persian text instances (single sentences, sentence pairs, or query-document pairs) depending on the specific task.

Output: Dense vector embeddings for each input text, which are subsequently processed by task-specific heads or similarity functions to produce predictions or ranked lists.

Scoring recipe

def compute_metric(predictions, gold, task_type):
    if task_type in ['classification', 'pair_classification', 'retrieval', 'summary_retrieval']:
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type in ['sts', 'ordinal_classification']:
        return pearsonr(predictions, gold).statistic
    elif task_type == 'reranking':
        return ndcg_at_k(gold, predictions, k=10)

Common pitfalls

• Confusing the synthetic data quality evaluation (human annotation agreement/correlation) with the actual model embedding performance evaluation.
• Using English baselines or metrics without verifying Persian script/tokenization compatibility, as translation quality directly impacts retrieval recall.
• Assuming all datasets use accuracy; STS and ordinal classification tasks require correlation coefficients instead.

Evidence (verbatim from paper)

For the other datasets, we used accuracy as the evaluation metric. Additionally, for retrieval datasets, each sample for tagging consisted of a pair: a query and a related document. The calculated accuracy indicates the percentage of samples where the query and document are indeed related.

Citation

@misc{zinvandi2025famteb,
  title={FaMTEB: Massive Text Embedding Benchmark in Persian Language},
  author={Zinvandi et al. (2025)},
  year={2025},
  note={arXiv:2502.11571}
}

• arXiv: 2502.11571