# Tanq Eval

> Evaluates open-domain, multi-hop question answering by requiring models to aggregate data from multiple sources and generate structured answer tables. It probes capabilities in multi-hop reasoning, data normalization, unit conversion, and table construction. Use when the user wants to benchmark on TANQ, or asks about evaluating this task. Reports F1.

- Skill: `qhjqhj00/tanq-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/tanq-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/tanq-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/tanq-eval

---


# tanq-eval

> TANQ: An open domain dataset of table answered questions — Akhtar et al. (2024) (arXiv:2405.07765, 2024)

## What this evaluates

Evaluates open-domain, multi-hop question answering by requiring models to aggregate data from multiple sources and generate structured answer tables. It probes capabilities in multi-hop reasoning, data normalization, unit conversion, and table construction.

## Datasets

- **TANQ** — total 1074; splits: test (1074); repo https://github.com/google-deepmind/tanq

## Metrics

- `F1` **(primary)** — range: [0, 100]
  - F1 = 2 * (Precision * Recall) / (Precision + Recall), computed at the cell level over the generated answer table compared to the gold table.

## Input / output format

**Input**: A natural language question q and a set of supporting documents D (oracle setting) or retrieved documents D' (open book/closed book settings).

**Output**: A structured answer table t with n rows and m columns, where each cell contains an extracted or derived entity/value.

## Scoring recipe

```python
def compute_f1(pred_table, gold_table):
    pred_cells = {cell for row in pred_table for cell in row}
    gold_cells = {cell for row in gold_table for cell in row}
    tp = len(pred_cells & gold_cells)
    fp = len(pred_cells - gold_cells)
    fn = len(gold_cells - pred_cells)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Column ordering in the answer table does not affect the F1 score but can confuse models and human evaluators.
- Some questions partially reveal the answer within the prompt, making the task easier than intended.
- Ambiguous or unclear relations in questions can lead to inconsistent cell extraction.

## Evidence (verbatim from paper)

> TANQ evaluates the capability to answer open domain, multi-hop questions by aggregating data and generating answer tables. ... resulting in a test set of 1,074 TANQ samples for evaluation. ... Table 5: Baseline performance by question type. For all question types, we observe Gemini Flash (60.7 F1) and PaLM-2 (47.6 F1) to outperform other baselines in oracle and closed book setting respectively, lagging 12.3 and 25.4 points behind the human baseline of 73.0.

## Citation

```bibtex
@misc{akhtar2024tanq,
  title={TANQ: An open domain dataset of table answered questions},
  author={Akhtar et al. (2024)},
  year={2024},
  note={arXiv:2405.07765}
}
```

- arXiv: 2405.07765