Mt Raig Eval

mt-raig-eval

MT-RAIG: Novel Benchmark and Evaluation Framework for Retrieval-Augmented Insight Generation over Multiple Tables — Kwangwook Seo, Donguk Kwon, Dongha Lee (2025) (arXiv:2502.11735, 2025)

What this evaluates

This benchmark evaluates retrieval-augmented insight generation over multiple tables. It requires models to retrieve relevant tables from a database and synthesize multi-hop insights across them, assessing both faithfulness and completeness of the generated reasoning.

Datasets

• MT-RAIG Bench — total ?; splits: (unstated)

Metrics

• MT-RAIG Eval (primary) — range: percent
  • Decomposes generated insights into table-aware and question-aware components. Auto-evaluation scores are normalized to a human preference scale of [-1, 0, 1], and the metric reports the Pearson correlation with human preference ratings.

Input / output format

Input: Natural language query/question and a database of tables for retrieval; for generation, the query and retrieved tables.

Output: A synthesized multi-hop insight text that answers the query using information from multiple tables.

Scoring recipe

def mt_raig_eval(predictions, golds, human_preferences):
    # Normalize auto-evaluation score differences to [-1, 0, 1]
    normalized_scores = normalize_to_preference_scale(predictions, golds)
    # Compute Pearson correlation with human ratings
    faithfulness_corr = pearson_correlation(normalized_scores['faithfulness'], human_preferences['faithfulness'])
    completeness_corr = pearson_correlation(normalized_scores['completeness'], human_preferences['completeness'])
    return faithfulness_corr, completeness_corr

Common pitfalls

• Table-specific embedding models underperform general text-based embeddings for this task.
• Increasing the number of retrieved tables (k) beyond a certain threshold introduces noise, causing faithfulness to drop while completeness remains stable.
• Existing TQA methods specialize in single-table factoid extraction and struggle with multi-table reasoning compared to general-purpose LLMs.

Evidence (verbatim from paper)

For scoring, we follow the setting of Ru et al. ([2024]) to normalize each auto-evaluation score difference to the human preference scale of [-1,0,1] and then measure the Pearson correlation with human preference ratings. Additionally, we report the correlation between the human evaluators as the upper bound. From the results in Table[5], we can observe that MT-RAIG Eval achieves the highest correlation with human preference ratings across both dimensions, demonstrating its reliability over baseline methods for evaluating MT-RAIG Bench.

Citation

@misc{seo2025mt_raig,
  title={MT-RAIG: Novel Benchmark and Evaluation Framework for Retrieval-Augmented Insight Generation over Multiple Tables},
  author={Kwangwook Seo, Donguk Kwon, Dongha Lee (2025)},
  year={2025},
  note={arXiv:2502.11735}
}

• arXiv: 2502.11735