Messirve Eval

messirve-eval

MessIRve: A Large-Scale Spanish Information Retrieval Dataset — Valentini et al. (2024) (arXiv:2409.05994, 2024)

What this evaluates

Evaluates information retrieval models on a large-scale, dialectally diverse Spanish dataset. It probes the ability of lexical and dense retrieval models to rank relevant Wikipedia documents for real-world Spanish search queries without fine-tuning.

Datasets

• MessIRve — total ?; splits: test (-1)

Metrics

• Recall@100 — range: [0, 1]
  • The fraction of relevant documents within the top 100 results, averaged over all queries.
• nDCG@10 (primary) — range: [0, 1]
  • The normalized Discounted Cumulative Gain. It compares the rank of the top 10 results to the ideal ranking where relevant documents are ranked higher. It is averaged over all queries.

Input / output format

Input: Query string and a corpus of Wikipedia documents (with titles appended to document text before retrieval).

Output: Ranked list of document IDs/URLs for each query.

Scoring recipe

def recall_at_k(retrieved, relevant, k=100):
    return len(set(retrieved[:k]) & set(relevant)) / len(relevant)

def ndcg_at_k(retrieved, rel_scores, k=10):
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(rel_scores[:k]))
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(sorted(rel_scores, reverse=True)[:k]))
    return dcg / idcg if idcg > 0 else 0.0

# Average over all queries
recall_avg = mean(recall_at_k(q_ret, q_rel) for q in queries)
ndcg_avg = mean(ndcg_at_k(q_ret, q_scores) for q in queries)

Common pitfalls

• Models are evaluated in a strict zero-shot setting; fine-tuning on the dataset is explicitly excluded.
• Wikipedia article titles are appended to document text before retrieval, which is a specific preprocessing step that affects baseline performance.
• Evaluation scores are averaged over all queries, though per-dialect partitions (e.g., ar, mx, bo, gt) are also reported.

Evidence (verbatim from paper)

We use two standard metrics to measure retrieval performance: Recall@100: the fraction of relevant documents within the top 100 results, averaged over all queries. nDCG@10: the normalized Discounted Cumulative Gain. It compares the rank of the top 10 results to the ideal ranking where relevant documents are ranked higher. It is averaged over all queries.

Citation

@misc{valentini2024messirve,
  title={MessIRve: A Large-Scale Spanish Information Retrieval Dataset},
  author={Valentini et al. (2024)},
  year={2024},
  note={arXiv:2409.05994}
}

• arXiv: 2409.05994