qhjqhj00 visual-tableqa-eval
Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images — Lompo et al. (2025) (arXiv:2509.07966, 2025)
What this evaluates
Probes multimodal visual reasoning capabilities over complex, LaTeX-rendered table images. It specifically tests multi-step inference, structural layout understanding, and the ability to extract and reason over tabular data from visual inputs rather than raw text.
Datasets
- Visual-TableQA — total 6000; splits: test (-1), train (-1); repo https://github.com/AI-4-Everyone/Visual-TableQA
Metrics
Relaxed Accuracy(primary) — range: percent- Fraction of correctly answered questions where the predicted answer matches the ground truth within a small numerical or semantic tolerance. The paper notes this metric typically inflates true accuracy by ~5%, which is subtracted for cross-benchmark ranking comparisons.
Input / output format
Input: Image-question pairs formatted into a unified prompt with a system message designed to elicit chain-of-thought reasoning. A LaTeX-code variant (Visual-TableQA-CIT) is also supported.
Output: Free-form text response containing the predicted answer and reasoning steps.
Scoring recipe
def compute_relaxed_accuracy(predictions, gold_answers, jury_models):
correct_count = 0
for pred, gold in zip(predictions, gold_answers):
votes = sum(1 for model in jury_models if model_judge(pred, gold, relaxed=True))
if votes > len(jury_models) / 2:
correct_count += 1
return (correct_count / len(predictions)) * 100
Common pitfalls
- Comparing raw Relaxed Accuracy scores across benchmarks without applying the paper's recommended 5% inflation adjustment for fair ranking.
- Confusing the image-based evaluation (Visual-TableQA) with the LaTeX-code variant (Visual-TableQA-CIT), which yields systematically higher scores (~+6.26%) and tests different capabilities.
- Assuming fine-tuning on this dataset guarantees transfer gains to other benchmarks; the paper shows transfer is asymmetric (e.g., fine-tuning on ReachQA degrades Visual-TableQA performance).
Evidence (verbatim from paper)
Performance metrics are measured using Relaxed Accuracy, which allows for small numerical deviations in the predicted answers. We assume that this accuracy inflates the actual accuracy by at least 5%. This margin is subtracted when selecting the best-performing results, which are shown in bold.
Citation
@misc{lompo2025visualtableqa,
title={Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images},
author={Lompo et al. (2025)},
year={2025},
note={arXiv:2509.07966}
}
- arXiv: 2509.07966