Granite R2 Retrieval Eval

granite-r2-retrieval-eval

Granite Embedding R2 Models — Awasthy et al. (2025) (arXiv:2508.21085, 2025)

What this evaluates

This evaluation protocol assesses the retrieval and reranking capabilities of encoder-based embedding models across diverse domains including general text, code, long documents, tables, and multi-turn conversations. It also measures encoding speed to evaluate efficiency in large-scale document ingestion pipelines.

Datasets

• MTEB-v2 — total ?; splits: test (-1)
• BEIR — total ?; splits: test (-1)
• COIR — total ?; splits: test (-1)
• MLDR — total ?; splits: test (-1)
• LongEmbed — total ?; splits: test (-1)
• Table IR — total ?; splits: test (-1)
• MT-RAG — total ?; splits: test (-1)
• IBM Documentation — total 23000; splits: test (23000)
• Miracl — total ?; splits: test (-1)

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of relevance scores up to position 10, normalized by the ideal DCG@10. Standard convention for ranking retrieval tasks.
• Recall@5 — range: [0, 1]
  • Fraction of relevant documents retrieved within the top 5 results. Used for MTRAG and specific Table IR tasks.
• Match@5 — range: [0, 1]
  • Exact match rate for the top 5 retrieved documents. Used for two Table IR tasks.
• Accuracy@1 — range: [0, 1]
  • Proportion of queries where the single top-ranked document is correct. Used for two LongEmbed datasets.
• Encoding Speed (Docs/s) — range: other
  • Number of documents processed per second on a single Nvidia H100 GPU with batch size 128. Documents are chunked to 512 tokens with 100-token overlap.

Input / output format

Input: Query and document pairs (or chunks). For retrieval, max sequence length is 8192 tokens. For speed benchmark, documents are chunked into 512 tokens with 100-token overlap. For reranking, queries are truncated to 64 tokens and top-20 retrieved documents are provided.

Output: Dense embedding vectors (768-dim for base models, 384-dim for small models) or reranking scores/rankings for the candidate documents.

Scoring recipe

def compute_avg_metric(predictions, gold, dataset_name):
    if dataset_name in ['MTRAG', 'NQTables', 'OTT-QA', 'MultiHierTT']:
        return recall_at_k(predictions, gold, k=5)
    elif dataset_name in ['AIT-QA']:
        return match_at_k(predictions, gold, k=5)
    elif dataset_name in ['LongEmbed_subset_1', 'LongEmbed_subset_2']:
        return accuracy_at_k(predictions, gold, k=1)
    else:
        return ndcg_at_k(predictions, gold, k=10)

def benchmark_score(predictions_map, gold_map, benchmark):
    scores = []
    for task in benchmark.tasks:
        scores.append(compute_avg_metric(predictions_map[task], gold_map[task], task))
    return sum(scores) / len(scores)

Common pitfalls

• Different benchmarks use different primary metrics (NDCG@10, Recall@5, Match@5, Accuracy@1); averaging requires mapping each dataset to its correct metric.
• Speed evaluation uses a fixed 512-token chunk size with 100-token overlap on a single H100 GPU with batch size 128, which may not generalize to other hardware or chunking strategies.
• Reranker evaluation is conditioned on top-20 documents retrieved by specific Granite embedding models, not a standard open retriever, limiting direct comparison with other rerankers.

Evidence (verbatim from paper)

Table 2: Retrieval Performance. Average scores are reported for benchmarks, with the number of tasks indicated in parentheses. MTRAG shows Recall@5. Two datasets in LongEmbed use Accuracy@1. Three tasks in Table IR use Recall@5 and two use Match@5. All other scores are average NDCG@10. Complete breakdown of scores is provided in Appendix [D].

Citation

@misc{awasthy2025graniteembeddingr2,
  title={Granite Embedding R2 Models},
  author={Awasthy et al. (2025)},
  year={2025},
  note={arXiv:2508.21085}
}

• arXiv: 2508.21085