Iclerb Eval

iclerb-eval

ICLERB: In-Context Learning Embedding and Reranker Benchmark — Al Ghossein et al. (2024) (arXiv:2411.18947, 2024)

What this evaluates

Evaluates embedding models and rerankers on their ability to retrieve contextually useful documents for In-Context Learning (ICL) tasks. It measures retrieval effectiveness by ranking candidate documents based on their utility in improving downstream LLM accuracy, rather than relying solely on semantic similarity.

Datasets

• TruthfulQA — total ?; splits: test (-1)
• Emotion — total ?; splits: test (-1)
• ProductER — total ?; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It measures the quality of a ranked list by applying a logarithmic discount to the relevance score of each retrieved document, normalized by the ideal DCG@10. Values range from 0 to 1, where 1 indicates a perfect ranking.

Input / output format

Input: A query representing an ICL task or prompt, paired with a pool of candidate documents or context snippets to be retrieved and ranked.

Output: A ranked list of candidate documents, typically evaluated on the top-10 results.

Scoring recipe

def compute_ndcg_at_10(relevance_scores, k=10):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal_scores = sorted(relevance_scores, reverse=True)
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_scores[:k]))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Rerankers fine-tuned for general semantic similarity or web search may underperform on ICL tasks compared to embedding models.
• Larger model size does not guarantee better ICL retrieval performance; smaller models can significantly outperform larger counterparts.

Evidence (verbatim from paper)

Table 2 presents the ICLERB results, sorted by nDCG@10. The top performers on the benchmark are cm-rerank-mxbai-rlaif-v0.1 (our proposed model, detailed in Section 5), BAAI's bge-en-icl, and NVIDIA's NV-Embed-v2.

Citation

@misc{alghossein2024iclerb,
  title={ICLERB: In-Context Learning Embedding and Reranker Benchmark},
  author={Al Ghossein et al. (2024)},
  year={2024},
  note={arXiv:2411.18947}
}

• arXiv: 2411.18947