Table QA Eval

table-qa-eval

MATA: Multi-Agent Framework for Reliable and Flexible Table Question Answering — Hyeon et al. (2026) (arXiv:2602.09642, 2026)

What this evaluates

Evaluates a model's ability to answer questions about tabular data using various reasoning strategies. It probes factual retrieval, numerical reasoning, and complex multi-step table understanding across different difficulty levels.

Datasets

• Penguins in a Table — total ?; splits: test (-1)
• TableBench — total ?; splits: test (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • 1 if the predicted answer string exactly matches the ground truth string, else 0.
• Fuzzy matching — range: [0, 1]
  • Textual similarity score based on Levenshtein distance, computed using the fuzzywuzzy library.
• SQuAD-style token-level F1 score — range: [0, 1]
  • Token-level overlap between prediction and ground truth, calculated as the harmonic mean of precision and recall at the token level.

Input / output format

Input: A table (in tabular format) and a natural language question.

Output: A single textual answer string.

Scoring recipe

def score(prediction, ground_truth):
    em = 1.0 if prediction.strip() == ground_truth.strip() else 0.0
    fuzzy = fuzz.ratio(prediction, ground_truth) / 100.0
    pred_tokens = set(prediction.lower().split())
    gt_tokens = set(ground_truth.lower().split())
    if not pred_tokens and not gt_tokens:
        f1 = 1.0
    elif not pred_tokens or not gt_tokens:
        f1 = 0.0
    else:
        prec = len(pred_tokens & gt_tokens) / len(pred_tokens)
        rec = len(pred_tokens & gt_tokens) / len(gt_tokens)
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return {'EM': em, 'Fuzzy': fuzzy, 'F1': f1}

Common pitfalls

• Exact Match (EM) is overly strict for TableQA and fails to capture semantically correct but syntactically different answers.
• Baselines like TabLaP and MixSC were originally tuned for closed-source GPT models and show significant performance drops when applied to open-source or smaller LLMs.
• Excessive LLM inference steps can degrade performance on simple tasks, so limiting reasoning paths is sometimes beneficial.

Evidence (verbatim from paper)

First, we used fuzzy matching222https://pypi.org/project/fuzzywuzzy/, a metric widely adopted in studies King and Flanigan ([2024]); Cheng et al. ([2024]); Nekvinda and Dušek ([2021]) to measure textual similarity based on Levenshtein distance Levenshtein ([1966]). Second, we adopted the SQuAD-style token-level F1 score Rajpurkar et al. ([2016]), which evaluates token-level overlap between the prediction and the ground truth. By incorporating these two metrics, we complement the strictness of EM with more flexible and nuanced evaluations.

Citation

@misc{hyeon2026mata,
  title={MATA: Multi-Agent Framework for Reliable and Flexible Table Question Answering},
  author={Hyeon et al. (2026)},
  year={2026},
  note={arXiv:2602.09642}
}

• arXiv: 2602.09642