Tanq Eval

tanq-eval

TANQ: An open domain dataset of table answered questions — Akhtar et al. (2024) (arXiv:2405.07765, 2024)

What this evaluates

Evaluates open-domain, multi-hop question answering by requiring models to aggregate data from multiple sources and generate structured answer tables. It probes capabilities in multi-hop reasoning, data normalization, unit conversion, and table construction.

Datasets

• TANQ — total 1074; splits: test (1074); repo https://github.com/google-deepmind/tanq

Metrics

• F1 (primary) — range: [0, 100]
  • F1 = 2 * (Precision * Recall) / (Precision + Recall), computed at the cell level over the generated answer table compared to the gold table.

Input / output format

Input: A natural language question q and a set of supporting documents D (oracle setting) or retrieved documents D' (open book/closed book settings).

Output: A structured answer table t with n rows and m columns, where each cell contains an extracted or derived entity/value.

Scoring recipe

def compute_f1(pred_table, gold_table):
    pred_cells = {cell for row in pred_table for cell in row}
    gold_cells = {cell for row in gold_table for cell in row}
    tp = len(pred_cells & gold_cells)
    fp = len(pred_cells - gold_cells)
    fn = len(gold_cells - pred_cells)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Column ordering in the answer table does not affect the F1 score but can confuse models and human evaluators.
• Some questions partially reveal the answer within the prompt, making the task easier than intended.
• Ambiguous or unclear relations in questions can lead to inconsistent cell extraction.

Evidence (verbatim from paper)

TANQ evaluates the capability to answer open domain, multi-hop questions by aggregating data and generating answer tables. ... resulting in a test set of 1,074 TANQ samples for evaluation. ... Table 5: Baseline performance by question type. For all question types, we observe Gemini Flash (60.7 F1) and PaLM-2 (47.6 F1) to outperform other baselines in oracle and closed book setting respectively, lagging 12.3 and 25.4 points behind the human baseline of 73.0.

Citation

@misc{akhtar2024tanq,
  title={TANQ: An open domain dataset of table answered questions},
  author={Akhtar et al. (2024)},
  year={2024},
  note={arXiv:2405.07765}
}

• arXiv: 2405.07765