T2 Ragbench Eval

t2-ragbench-eval

T$^2$-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation — Strich et al. (2025) (arXiv:2506.12071, 2025)

What this evaluates

Evaluates Retrieval-Augmented Generation (RAG) systems on their ability to retrieve relevant text-and-table contexts from financial reports and perform numerical reasoning to answer questions. It measures both retrieval effectiveness and the accuracy of the generated numerical answers.

Datasets

• T2-RAGBench — total 32908; splits: test (-1)

Metrics

• Number Match (NM) (primary) — range: percent
  • Percentage of questions where the predicted numerical value matches the gold answer within a relative tolerance of epsilon=1e-2. Non-numeric or out-of-tolerance predictions are scored as 0.
• MRR@3 (primary) — range: [0, 1]
  • Mean Reciprocal Rank of the first relevant document within the top-3 retrieved results. Calculated as the average of 1/rank for each query.

Input / output format

Input: A question and the top-3 retrieved documents (in markdown format) passed to the generator.

Output: A numerical answer or text containing the predicted value.

Scoring recipe

def score_nm(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        try:
            if abs(float(pred) - float(gold)) / abs(float(gold)) <= 1e-2:
                correct += 1
        except:
            pass
    return correct / len(golds)

def score_mrr(retrieval_ranks, k=3):
    scores = []
    for rank in retrieval_ranks:
        if rank <= k:
            scores.append(1.0 / rank)
        else:
            scores.append(0.0)
    return sum(scores) / len(scores)

Common pitfalls

• Limiting retrieval to top-3 documents is critical; using more degrades LLM performance and is deemed impractical.
• Summarization-based augmentation often drops Number Match because essential numerical details are lost during condensation.
• Cross-encoder rerankers trained on text-only corpora underperform on text-and-table data compared to hybrid BM25.

Evidence (verbatim from paper)

We use Number Match and MRR@k as our main metrics as defined in Section[3], but also report Recall@1 (R@1) and Recall@3 (R@3) in the Appendix[I] for better comparability and transparency. Number Match evaluates if a numerical prediction closely matches the gold numerical answer. It compares predicted and ground truth values using relative tolerance ($\epsilon=1\mathrm{e}{-2}$), accounting for scale invariance. Non-numeric predictions or mismatches are considered incorrect. For MRR we choose $k=3$, what measures whether the first relevant document appears in the top-3 retrieved results, rewarding higher ranks.

Citation

@misc{strich2025t2ragbench,
  title={T$^2$-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation},
  author={Strich et al. (2025)},
  year={2025},
  note={arXiv:2506.12071}
}

• arXiv: 2506.12071