Mmarco Passage Ranking Eval

mmarco-passage-ranking-eval

mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset — Bonifacio et al. (2021) (arXiv:2108.13897, 2021)

What this evaluates

Evaluates multilingual passage retrieval models on translated versions of the MS MARCO dataset and the Mr. TyDi dataset. It probes the ability of dense retrieval and reranking models to handle cross-lingual and zero-shot retrieval scenarios, as well as the impact of translation quality on retrieval effectiveness.

Datasets

• mMARCO — total ?; splits: development (-1); repo https://github.com/unicamp-dl/mMARCO
• Mr. TyDi — total ?; splits: test (-1)

Metrics

• MRR@10 (primary) — range: [0, 1]
  • Mean Reciprocal Rank of the first relevant document within the top 10 retrieved passages. Calculated as the average of 1/rank for each query where rank is the position of the first relevant result.
• Recall@1000 — range: [0, 1]
  • Fraction of all relevant passages retrieved within the top 1000 results. Calculated as the size of the intersection between retrieved and relevant sets divided by the size of the relevant set.
• Recall@100 — range: [0, 1]
  • Fraction of all relevant passages retrieved within the top 100 results. Calculated as the size of the intersection between retrieved and relevant sets divided by the size of the relevant set.

Input / output format

Input: Query and passage text in a specific language.

Output: Ranked list of passages for each query.

Scoring recipe

def compute_mrr_at_k(predictions, gold, k=10):
    scores = []
    for pred_list, gold_set in zip(predictions, gold):
        for rank, doc_id in enumerate(pred_list[:k], 1):
            if doc_id in gold_set:
                scores.append(1.0 / rank)
                break
        else:
            scores.append(0.0)
    return sum(scores) / len(scores)

def compute_recall_at_k(predictions, gold, k=1000):
    scores = []
    for pred_list, gold_set in zip(predictions, gold):
        retrieved = set(pred_list[:k])
        scores.append(len(retrieved & gold_set) / len(gold_set))
    return sum(scores) / len(scores)

Common pitfalls

• Queries and passages are translated independently, causing lexical mismatches that penalize lexical models like BM25.
• Zero-shot evaluation tests models on languages completely absent from the fine-tuning data, which can yield lower performance than in-domain evaluation.
• The correlation between translation quality (BLEU) and retrieval effectiveness (MRR@10) is weak (R^2 ≈ 0.33), so high BLEU does not guarantee better retrieval.

Evidence (verbatim from paper)

We report MRR@10, which is the official metric of the MS MARCO passage dataset, as well as recall@1000.

Citation

@misc{bonifacio2021mmarco,
  title={mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset},
  author={Bonifacio et al. (2021)},
  year={2021},
  note={arXiv:2108.13897}
}

• arXiv: 2108.13897