Wikitq Eval

wikitq-eval

Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding — Wang et al. (2024) (arXiv:2401.04398, 2024)

What this evaluates

Evaluates a model's ability to perform complex tabular reasoning to answer open-ended questions based on a provided table. It probes the model's capacity to extract, aggregate, and filter information from structured data to produce short text span answers.

Datasets

• WikiTQ — total ?; splits: test (-1)

Metrics

• denotation accuracy (primary) — range: [0, 1]
  • Exact match between the predicted answer and the set of gold-standard answers. A prediction is correct only if it exactly matches one of the valid answers.

Input / output format

Input: A tabular dataset and a natural language question.

Output: A short text span representing the answer.

Scoring recipe

def score(pred, gold):
    return 1.0 if pred in gold else 0.0
accuracy = sum(score(p, g) for p, g in zip(predictions, golds)) / len(golds)

Common pitfalls

• Denotation accuracy requires exact string match against a predefined set of valid answers, not fuzzy or semantic similarity.
• The official evaluation pipeline must be used to handle case sensitivity and whitespace normalization correctly.

Evidence (verbatim from paper)

For WikiTQ evaluation, we use the official denotation accuracy*(Pasupat & Liang, 2015)*, and for TabFact, we employ the binary classification accuracy.

Citation

@misc{wang2024chainoftable,
  title={Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2401.04398}
}

• arXiv: 2401.04398