Wikitablequestions Eval

wikitablequestions-eval

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data — Pengcheng Yin et al. (2020) (arXiv:2005.08314, 2020)

What this evaluates

Evaluates a model's ability to perform semantic parsing over semi-structured tabular data by generating database queries or answers from natural language questions. It measures how well the model aligns textual utterances with table schemas and content to retrieve correct results.

Datasets

• WIKITABLEQUESTIONS — total ?; splits: dev (-1), test (-1)

Metrics

• execution accuracy (primary) — range: percent
  • The percentage of test instances where the generated database query produces the exact same result as the gold query when executed against the target table.

Input / output format

Input: A natural language question (utterance) paired with a database table represented via row linearization and content snapshots (sampled rows).

Output: A database query or direct answer string.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, golds):
    if execute_query(pred, table) == execute_query(gold, table):
        correct += 1
return (correct / len(predictions)) * 100

Common pitfalls

• Execution accuracy requires actually running the generated query against the table, not just string-matching the SQL.
• The paper evaluates using 10 random runs and reports mean ± std and best dev-tuned test scores, which can inflate performance if not properly tracked.

Evidence (verbatim from paper)

Evaluation Metrics As standard, we report execution accuracy on WIKITABLEQUESTIONS and exact-match accuracy of DB queries on SPIDER.

Citation

@misc{yin2020tabert,
  title={TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data},
  author={Pengcheng Yin et al. (2020)},
  year={2020},
  note={arXiv:2005.08314}
}

• arXiv: 2005.08314