Low Resource Embedding Eval

low-resource-embedding-eval

Bootstrapping Embeddings for Low Resource Languages — Basoz et al. (2026) (arXiv:2603.01732, 2026)

What this evaluates

Evaluates sentence embedding quality for low-resource languages trained on synthetic triplet data. It probes cross-lingual semantic similarity and information retrieval capabilities by benchmarking against human-annotated and unsupervised baselines without requiring target-language training data.

Datasets

• Ousidhoum STS/STR — total ?; splits: test (-1)
• MTEB Retrieval (Low-Resource Subset) — total ?; splits: test (-1)

Metrics

• Spearman's correlation (primary) — range: [-1, 1]
  • Spearman rank correlation coefficient between predicted similarity scores and ground-truth similarity ratings. Measures the monotonic relationship between predictions and human/anchor labels.
• Recall@10 — range: [0, 1]
  • Fraction of queries where the relevant document appears in the top 10 retrieved results.

Input / output format

Input: Sentence pairs for STS tasks; query-document pairs for retrieval tasks, all in target low-resource languages.

Output: Continuous similarity score for STS; ranked list of document IDs with scores for retrieval.

Scoring recipe

def compute_metrics(pred_scores, gold_scores, queries, retrieved_ids, gold_doc_ids):
    # STS
    spearman_corr = scipy.stats.spearmanr(pred_scores, gold_scores).correlation
    # Retrieval
    recall_at_10 = sum(1 for q, g in zip(queries, gold_doc_ids) if g in retrieved_ids[q][:10]) / len(queries)
    # Aggregate
    return spearman_corr, recall_at_10
# Report mean ± std over 4 random seeds

Common pitfalls

• Evaluating zero-shot cross-lingual transfer (trained on English, tested on target) as if it were direct target-language training.
• Ignoring seed variance; the protocol requires reporting mean ± standard deviation over four random seeds.
• Assuming synthetic triplet generation requires an LLM fine-tuned on the target language; the protocol uses cross-lingual adaptation (XL-LoRA) without target-language training data.

Evidence (verbatim from paper)

We evaluate performance using both STS/STR tasks from Ousidhoum et al. (2024) and a subset of retrieval tasks from MTEB Muennighoff et al. (2022) specifically focusing on low resource languages. Table 1: Embedding performance on STS tasks (Spearman’s correlation). Figure 3: Retrieval results across multiple benchmarks. Results are averaged across backbones (XLM-R and mmBERT) and across languages. Metric is recall@10.

Citation

@misc{basoz2026bootstrapping,
  title={Bootstrapping Embeddings for Low Resource Languages},
  author={Basoz et al. (2026)},
  year={2026},
  note={arXiv:2603.01732}
}

• arXiv: 2603.01732