Mrtydi Eval

mrtydi-eval

Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval — Zhang et al. (2021) (arXiv:2108.08787, 2021)

What this evaluates

Evaluates mono-lingual dense retrieval models across eleven typologically diverse languages by measuring their ability to rank relevant Wikipedia passages for given questions. It probes zero-shot cross-lingual generalization and the effectiveness of sparse-dense hybrid retrieval compared to strong sparse baselines.

Datasets

• Mr. TYDI v1.1 — total ?; splits: test (-1); repo https://github.com/castorini/mr.tydi

Metrics

• MRR@100 (primary) — range: [0, 1]
  • Mean Reciprocal Rank at cutoff 100. For each query, compute 1/rank of the first relevant passage in the top-100 retrieved results. Average across all queries in the test set.
• Recall@100 — range: [0, 1]
  • Recall at cutoff 100. Proportion of queries for which at least one relevant passage appears in the top-100 retrieved results.

Input / output format

Input: A question (query) in one of eleven languages, to be matched against a full Wikipedia corpus in the same language.

Output: A ranked list of passages (each containing a Wikipedia article title and passage text) returned by the retrieval model.

Scoring recipe

def compute_metrics(retrieved_passages, gold_passages, k=100):
    if not gold_passages: return 0.0, 0.0
    rr = 0.0
    found = False
    for i, pid in enumerate(retrieved_passages[:k]):
        if pid in gold_passages:
            rr = 1.0 / (i + 1)
            found = True
            break
    recall = 1.0 if found else 0.0
    return rr, recall
# Average across all queries in the test set

Common pitfalls

• Absolute metric scores vary significantly across languages due to differences in question phrasing and corpus characteristics, making direct cross-lingual comparison misleading without normalization.
• MRR@100 heavily penalizes queries where no relevant passage is retrieved in the top-100, which can mask a model's strong ranking ability for the subset of queries where it does find relevant results.
• Zero-shot dense retrieval models (e.g., mDPR trained on English NQ) often underperform strong sparse baselines (BM25) in non-English languages, highlighting distribution shift rather than model architecture flaws.

Evidence (verbatim from paper)

Table 2 reports results on the test set across all eleven languages; mean reciprocal rank (MRR) in the top table and recall in the bottom table, both at a cutoff of 100 hits; the final column reports the average across all languages.

Citation

@misc{zhang2021mrtydi,
  title={Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval},
  author={Zhang et al. (2021)},
  year={2021},
  note={arXiv:2108.08787}
}

• arXiv: 2108.08787