# Tqabench Eval

> Evaluates large language models' ability to perform multi-table question answering across varying context lengths (8K–64K tokens) and complex reasoning tasks. It probes cross-table inference, symbolic reasoning, and handling of real-world relational data without Wikipedia bias. Use when the user wants to benchmark on TQA-Bench, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/tqabench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/tqabench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/tqabench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/tqabench-eval

---


# tqabench-eval

> TQA-Bench: Evaluating LLMs for Multi-Table Question Answering with Scalable Context and Symbolic Extension — Qiu et al. (2024) (arXiv:2411.19504, 2024)

## What this evaluates

Evaluates large language models' ability to perform multi-table question answering across varying context lengths (8K–64K tokens) and complex reasoning tasks. It probes cross-table inference, symbolic reasoning, and handling of real-world relational data without Wikipedia bias.

## Datasets

- **TQA-Bench** — total ?; splits: test (-1); repo https://github.com/Relaxed-System-Lab/TQA-Bench

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact-match accuracy computed by comparing the model's predicted answer string against the ground truth answer generated via Python scripts. A prediction is correct if it exactly matches the gold answer after standard normalization.

## Input / output format

**Input**: Serialized multi-table database in Markdown format (including table names and foreign key relationships) concatenated with a natural language question. Context length varies from 8K to 64K tokens.

**Output**: Natural language answer string corresponding to the question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Models often fail when serialized tables exceed 8K tokens due to context window limits, despite the benchmark's scalable sampling design.
- Naive row sampling breaks foreign key referential integrity, invalidating cross-table queries and symbolic extensions.
- Symbolic augmentations require multi-step logical reasoning rather than simple pattern matching, causing significant performance drops on complex calculation tasks.

## Evidence (verbatim from paper)

> For each question category, we develop structured question templates that are augmented with symbolic extensions to assess reasoning capabilities beyond simple retrieval. These templates are paired with Python-based answer generation, enabling the automated creation of benchmark questions and ensuring scalability and reliability in task evaluation.

## Citation

```bibtex
@misc{qiu2024tqabench,
  title={TQA-Bench: Evaluating LLMs for Multi-Table Question Answering with Scalable Context and Symbolic Extension},
  author={Qiu et al. (2024)},
  year={2024},
  note={arXiv:2411.19504}
}
```

- arXiv: 2411.19504