Retrieval Rerank RAG Eval

retrieval-rerank-rag-eval

Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation — Abdelrahman Abdallah et al. (2025) (arXiv:2502.02464, 2025)

What this evaluates

Evaluates the effectiveness of sparse and dense retrieval models, re-rankers, and retrieval-augmented generation (RAG) pipelines on open-domain QA, multi-hop QA, and fact verification tasks.

Datasets

• Natural Questions — total ?; splits: test (-1)
• TriviaQA — total ?; splits: test (-1)
• HotpotQA — total ?; splits: test (-1)
• 2WikiMultiHopQA — total ?; splits: test (-1)
• ArchivalQA — total ?; splits: test (-1)
• MSMARCO — total ?; splits: test (-1)
• WebQuestions — total ?; splits: test (-1)
• PopQA — total ?; splits: test (-1)
• ChroniclingAmericaQA — total ?; splits: test (-1)

Metrics

• Top-k accuracy (primary) — range: percent
  • Measures the proportion of queries where the ground-truth document appears in the top-k retrieved results. Evaluated at k=1, 5, 10, 20, 50, 100.
• Exact match (EM) (primary) — range: percent
  • Measures the proportion of generated answers that exactly match the ground-truth reference string.
• Precision — range: percent
  • Standard retrieval precision metric.
• Recall — range: percent
  • Standard retrieval recall metric.
• Contains — range: percent
  • Binary metric indicating whether the ground-truth answer string is contained within the generated answer.
• F1 — range: percent
  • Harmonic mean of Precision and Recall.

Input / output format

Input: A query and a corpus of candidate passages (e.g., 100-word non-overlapping Wikipedia passages).

Output: A ranked list of retrieved documents or a generated answer string for RAG.

Scoring recipe

def score_topk_accuracy(predictions, gold, k):
    correct = 0
    for pred, gold_set in zip(predictions, gold):
        if any(g in pred[:k] for g in gold_set):
            correct += 1
    return correct / len(predictions)

def score_exact_match(predictions, gold):
    correct = 0
    for p, g in zip(predictions, gold):
        if p.strip() == g.strip():
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Datasets use different underlying corpora or splits, making direct comparison across studies difficult.
• Re-ranking experiments used top 100 retrieved documents, whereas original studies used 1,000, causing result differences.
• Some datasets (e.g., ChroniclingAmericaQA) do not report retrieval accuracy as a standard Top-k metric.

Evidence (verbatim from paper)

We evaluated retrieval performance on datasets such as Natural Questions (NQ), TriviaQA, HotpotQA, 2WikiMultiHopQA, and ArchivalQA, while re-ranking performance was analyzed on MSMARCO, WebQuestions, and PopQA. For retrieval evaluation, we measured Top-k accuracy at k=1, 5, 10, 20, 50, 100. Exact match (EM), Precision, Recall, Contains, and F1 were used as the primary evaluation metrics for QA tasks.

Citation

@misc{abdallah2025rankify,
  title={Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation},
  author={Abdelrahman Abdallah et al. (2025)},
  year={2025},
  note={arXiv:2502.02464}
}

• arXiv: 2502.02464