Miracl Eval

miracl-eval

M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation — Chen et al. (2024) (arXiv:2402.03216, 2024)

What this evaluates

Evaluates multi-lingual ad-hoc retrieval across 18 languages, testing a model's ability to match queries and passages in the same language using dense, sparse, and multi-vector embedding strategies.

Datasets

• MIRACL — total ?; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Measures the quality of the ranked list of retrieved passages relative to the ideal ranking, normalized by the ideal DCG@10.
• Recall@100 — range: [0, 1]
  • The proportion of relevant passages found within the top 100 retrieved results.

Input / output format

Input: A query and a corpus of passages, all presented in the same language.

Output: A ranked list of top-1000 candidates (for dense/sparse) or top-200 candidates (for multi-vector reranking), combined via weighted fusion for hybrid methods.

Scoring recipe

def compute_ndcg_at_10(retrieved_ids, relevant_ids):
    dcg = 0.0
    for i, doc_id in enumerate(retrieved_ids[:10]):
        if doc_id in relevant_ids:
            dcg += 1.0 / math.log2(i + 2)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_ids), 10)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• The paper uses different retrieval pipelines for dense/sparse (Faiss/Lucene top-1000) vs multi-vector (reranker top-200), which affects latency and recall.
• Hybrid retrieval weights (w1, w2, w3) are fixed per task (e.g., w1=1, w2=0.3 for MIRACL) rather than tuned per language.

Evidence (verbatim from paper)

We evaluate the multi-lingual retrieval performance with MIRACL, which consists of ad-hoc retrieval tasks in 18 languages. Each task is made up of query and passage presented in the same language. Following the official benchmark, we evaluate our method using Pyserini, and use nDCG@10 as the primary evaluation metric (Recall@100 is also measured and reported in Appendix[C.1]).

Citation

@misc{chen2024m3embedding,
  title={M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2402.03216}
}

• arXiv: 2402.03216