# Mt Raig Eval

> This benchmark evaluates retrieval-augmented insight generation over multiple tables. It requires models to retrieve relevant tables from a database and synthesize multi-hop insights across them, assessing both faithfulness and completeness of the generated reasoning. Use when the user wants to benchmark on MT-RAIG Bench, or asks about evaluating this task. Reports MT-RAIG Eval.

- Skill: `qhjqhj00/mt-raig-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mt-raig-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mt-raig-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mt-raig-eval

---


# mt-raig-eval

> MT-RAIG: Novel Benchmark and Evaluation Framework for Retrieval-Augmented Insight Generation over Multiple Tables — Kwangwook Seo, Donguk Kwon, Dongha Lee (2025) (arXiv:2502.11735, 2025)

## What this evaluates

This benchmark evaluates retrieval-augmented insight generation over multiple tables. It requires models to retrieve relevant tables from a database and synthesize multi-hop insights across them, assessing both faithfulness and completeness of the generated reasoning.

## Datasets

- **MT-RAIG Bench** — total ?; splits: (unstated)

## Metrics

- `MT-RAIG Eval` **(primary)** — range: percent
  - Decomposes generated insights into table-aware and question-aware components. Auto-evaluation scores are normalized to a human preference scale of [-1, 0, 1], and the metric reports the Pearson correlation with human preference ratings.

## Input / output format

**Input**: Natural language query/question and a database of tables for retrieval; for generation, the query and retrieved tables.

**Output**: A synthesized multi-hop insight text that answers the query using information from multiple tables.

## Scoring recipe

```python
def mt_raig_eval(predictions, golds, human_preferences):
    # Normalize auto-evaluation score differences to [-1, 0, 1]
    normalized_scores = normalize_to_preference_scale(predictions, golds)
    # Compute Pearson correlation with human ratings
    faithfulness_corr = pearson_correlation(normalized_scores['faithfulness'], human_preferences['faithfulness'])
    completeness_corr = pearson_correlation(normalized_scores['completeness'], human_preferences['completeness'])
    return faithfulness_corr, completeness_corr
```

## Common pitfalls

- Table-specific embedding models underperform general text-based embeddings for this task.
- Increasing the number of retrieved tables (k) beyond a certain threshold introduces noise, causing faithfulness to drop while completeness remains stable.
- Existing TQA methods specialize in single-table factoid extraction and struggle with multi-table reasoning compared to general-purpose LLMs.

## Evidence (verbatim from paper)

> For scoring, we follow the setting of Ru et al. ([2024]) to normalize each auto-evaluation score difference to the human preference scale of [-1,0,1] and then measure the Pearson correlation with human preference ratings. Additionally, we report the correlation between the human evaluators as the upper bound. From the results in Table[5], we can observe that MT-RAIG Eval achieves the highest correlation with human preference ratings across both dimensions, demonstrating its reliability over baseline methods for evaluating MT-RAIG Bench.

## Citation

```bibtex
@misc{seo2025mt_raig,
  title={MT-RAIG: Novel Benchmark and Evaluation Framework for Retrieval-Augmented Insight Generation over Multiple Tables},
  author={Kwangwook Seo, Donguk Kwon, Dongha Lee (2025)},
  year={2025},
  note={arXiv:2502.11735}
}
```

- arXiv: 2502.11735