Irsc Eval

irsc-eval

IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios — Lin et al. (2024) (arXiv:2409.15763, 2024)

What this evaluates

Evaluates embedding models on multilingual information retrieval tasks across five query types (query, title, part-of-paragraph, keyword, summary). It probes semantic comprehension and cross-lingual retrieval alignment in Retrieval-Augmented Generation (RAG) scenarios.

Datasets

• IRSC Benchmark — total 5000; splits: test (5000); repo https://github.com/Jasaxion/IRSC_Benchmark

Metrics

• r@10 (primary) — range: [0, 1]
  • Recall at 10: the proportion of relevant documents retrieved within the top 10 results out of the total number of relevant documents.
• m@10 — range: [0, 1]
  • Mean Average Precision at 10 (MAP@10): the average precision score calculated at the 10th retrieved document.
• n@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at 10 (NDCG@10): a measure of ranking quality that discounts the relevance of retrieved documents based on their position.

Input / output format

Input: A query string and a candidate document/paragraph (or a set of candidate documents) in English, Chinese, or mixed language.

Output: A relevance score for each query-document pair, or a ranked list of retrieved documents.

Scoring recipe

def compute_metrics(retrieved_docs, relevant_docs, k=10):
    retrieved = retrieved_docs[:k]
    rel_set = set(relevant_docs)
    r10 = len(set(retrieved) & rel_set) / max(len(rel_set), 1)
    hits = 0
    sum_prec = 0.0
    for i, doc in enumerate(retrieved):
        if doc in rel_set:
            hits += 1
            sum_prec += hits / (i + 1)
    m10 = sum_prec / max(len(rel_set), 1)
    dcg = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(retrieved) if doc in rel_set)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(rel_set), k)))
    n10 = dcg / max(idcg, 1e-9)
    return r10, m10, n10

Common pitfalls

• Cross-lingual tasks require explicit query translation before retrieval, which can introduce translation errors not inherent to the embedding model.
• Task complexity varies significantly; models perform well on Summary/Keyword tasks but struggle with Title/Part tasks due to differing semantic compression requirements.
• Metrics are averaged across five sub-tasks; reporting only the aggregate score masks task-specific weaknesses.

Evidence (verbatim from paper)

Specifically, BGE-M3 achieves the highest recall at 10 (r@10), mean average precision at 10 (m@10), and normalized discounted cumulative gain at 10 (n@10) in the Keywords, Title, Query, Part, and Summary categories. For instance, in the Keywords category, BGE-M3 has an impressive r@10 of 0.8668, m@10 of 0.8205, and n@10 of 0.8320.

Citation

@misc{lin2024irsc,
  title={IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios},
  author={Lin et al. (2024)},
  year={2024},
  note={arXiv:2409.15763}
}

• arXiv: 2409.15763