qhjqhj00 crosslingual-speech-text-retrieval-eval
Cross-lingual Matryoshka Representation Learning across Speech and Text — Sy et al. (2026) (arXiv:2602.19991, 2026)
What this evaluates
Evaluates cross-lingual speech-to-text retrieval and intent detection capabilities across multiple datasets, testing how well speech queries can retrieve relevant text documents or classify intents without intermediate ASR or translation pipelines.
Datasets
- Kallaama-Retrieval-Eval — total ?; splits: (unstated)
- Fleurs-Retrieval-Eval — total ?; splits: (unstated)
- Urban Bus — total ?; splits: test (-1)
- WolBanking77 — total ?; splits: (unstated)
Metrics
nDCG@5(primary) — range: percent- Normalized Discounted Cumulative Gain at k=5. Measures ranking quality by comparing the graded relevance of the top 5 retrieved documents to an ideal ranking.
nDCG@10— range: percent- Normalized Discounted Cumulative Gain at k=10. Same as nDCG@5 but evaluates the top 10 retrieved documents.
F1-Score— range: percent- Harmonic mean of precision and recall for intent detection and keyword spotting tasks.
Recall— range: percent- Proportion of correctly identified positive instances (keywords or intents) out of all actual positives.
Input / output format
Input: Speech audio query (and optionally text query for dual-encoder baselines) paired with a corpus of text documents for retrieval; or speech audio query for intent/keyword spotting.
Output: Ranked list of retrieved text documents (for retrieval tasks); predicted intent label or keyword match (for spotting/detection tasks).
Scoring recipe
def compute_ndcg_at_k(retrieved, gold, k):
dcg = sum(rel / log2(i + 2) for i, rel in enumerate(retrieved[:k]))
idcg = sum(rel / log2(i + 2) for i, rel in enumerate(sorted(gold, reverse=True)[:k]))
return (dcg / idcg) * 100 if idcg > 0 else 0.0
def compute_f1_recall(preds, golds):
tp = sum(p == g for p, g in zip(preds, golds))
fp = sum(p != g for p in preds if p not in golds)
fn = sum(g != p for g in golds if g not in preds)
prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
return f1 * 100, rec * 100
Common pitfalls
- Matryoshka embedding dimensionality significantly impacts performance, with lower dimensions degrading faster on low-quality speech (Fleurs) than high-quality speech (Kallaama).
- Pipelined baselines (ASR then text retrieval) suffer from transcription error propagation, which the direct speech-text models avoid.
- NLLB-LLM2Vec baseline uses a fixed 4096-dimension embedding, making direct comparison with variable-dimension Matryoshka models require careful normalization or dimension-matching.
Evidence (verbatim from paper)
We evaluate the trained models on Kallaama-Retrieval-Eval and Fleurs-Retrieval-Eval using nDCG, a standard metric to evaluate the ranking of recommender systems. nDCG@$k$ measures the ranking quality of the top $k$ retrieved documents by comparing their graded relevance to an ideal ranking, with higher scores indicating better alignment with the ground truth.
Citation
@misc{sy2026crosslingualmatryoshka,
title={Cross-lingual Matryoshka Representation Learning across Speech and Text},
author={Sy et al. (2026)},
year={2026},
note={arXiv:2602.19991}
}
- arXiv: 2602.19991