Swim Ir Eval

swim-ir-eval

Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval — Thakur et al. (2023) (arXiv:2311.05800, 2023)

What this evaluates

Evaluates multilingual dense retrieval models on cross-lingual and monolingual open retrieval tasks. It measures how effectively synthetic LLM-generated training data scales retrieval performance compared to human-labeled baselines across diverse languages and corpus sizes.

Datasets

• XOR-Retrieve — total 17000; splits: train (15000), dev (2000)
• MIRACL — total 101783; splits: train (88288), dev (13495)
• XTREME-UP — total 15985; splits: train (5280), test (10705)

Metrics

• Recall@mkt (primary) — range: [0, 1]
  • Computes the fraction of queries for which the minimal answer is contained within the top m thousand tokens of the retrieved passages. Evaluated at m=5 and m=2.
• nDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring ranking quality based on graded relevance judgments.
• Recall@100 — range: [0, 1]
  • Fraction of queries where the relevant passage appears in the top 100 retrieved results.
• MRR@10 — range: [0, 1]
  • Mean Reciprocal Rank at rank 10, averaging the inverse rank of the first relevant passage for each query.

Input / output format

Input: Query string (monolingual or multilingual) and a passage corpus (Wikipedia or TyDi-QA passages).

Output: Ranked list of passages (or top-k passages) for each query.

Scoring recipe

def compute_recall_mkt(retrieved_passages, gold_answer, k=5):
    token_limit = k * 1000
    acc_tokens = 0
    for p in retrieved_passages:
        acc_tokens += len(p.split())
        if acc_tokens >= token_limit:
            break
    return 1.0 if gold_answer in retrieved_passages else 0.0

Common pitfalls

• SWIM-X models were not optimized with hard negatives, unlike supervised baselines that use up to four hard negatives per pair.
• Corpus token limits vary by dataset (e.g., 100-word tokens for XOR-Retrieve, varying sizes for MIRACL), affecting Recall@mkt calculations.
• Translation script mismatches (e.g., Manipuri in 'Meitei' vs 'Bengali-Assamese' script) can cause evaluation failures if not handled.

Evidence (verbatim from paper)

We evaluate our models using recall at m kilo-tokens, i.e., Recall@mkt, which computes the fraction of queries for which the minimal answer is contained within the top m thousand tokens of the retrieved passages. Following prior work in Asai et al. (2021a), we evaluate our models at Recall@5kt and Recall@2kt.

Citation

@misc{thakur2023swimir,
  title={Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval},
  author={Thakur et al. (2023)},
  year={2023},
  note={arXiv:2311.05800}
}

• arXiv: 2311.05800