Tqabench Eval

tqabench-eval

TQA-Bench: Evaluating LLMs for Multi-Table Question Answering with Scalable Context and Symbolic Extension — Qiu et al. (2024) (arXiv:2411.19504, 2024)

What this evaluates

Evaluates large language models' ability to perform multi-table question answering across varying context lengths (8K–64K tokens) and complex reasoning tasks. It probes cross-table inference, symbolic reasoning, and handling of real-world relational data without Wikipedia bias.

Datasets

• TQA-Bench — total ?; splits: test (-1); repo https://github.com/Relaxed-System-Lab/TQA-Bench

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy computed by comparing the model's predicted answer string against the ground truth answer generated via Python scripts. A prediction is correct if it exactly matches the gold answer after standard normalization.

Input / output format

Input: Serialized multi-table database in Markdown format (including table names and foreign key relationships) concatenated with a natural language question. Context length varies from 8K to 64K tokens.

Output: Natural language answer string corresponding to the question.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Models often fail when serialized tables exceed 8K tokens due to context window limits, despite the benchmark's scalable sampling design.
• Naive row sampling breaks foreign key referential integrity, invalidating cross-table queries and symbolic extensions.
• Symbolic augmentations require multi-step logical reasoning rather than simple pattern matching, causing significant performance drops on complex calculation tasks.

Evidence (verbatim from paper)

For each question category, we develop structured question templates that are augmented with symbolic extensions to assess reasoning capabilities beyond simple retrieval. These templates are paired with Python-based answer generation, enabling the automated creation of benchmark questions and ensuring scalability and reliability in task evaluation.

Citation

@misc{qiu2024tqabench,
  title={TQA-Bench: Evaluating LLMs for Multi-Table Question Answering with Scalable Context and Symbolic Extension},
  author={Qiu et al. (2024)},
  year={2024},
  note={arXiv:2411.19504}
}

• arXiv: 2411.19504