Open Wikitable Eval

open-wikitable-eval

Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table — Kweon et al. (2023) (arXiv:2305.07288, 2023)

What this evaluates

Evaluates open-domain table retrieval and end-to-end question answering over complex table reasoning tasks. It probes a model's ability to retrieve relevant table segments from a corpus and then answer questions using either direct reading or SQL generation.

Datasets

• Open-WikiTable — total ?; splits: validation (-1), test (-1); repo https://github.com/seaon0042/Open_

Metrics

• Top-k table retrieval accuracy (primary) — range: [0, 1]
  • Percentage of queries where the ground-truth table segment appears in the top-k retrieved results.
• exact match (EM) accuracy — range: [0, 1]
  • For readers: exact string match between predicted and gold answer. For parsers: exact match on the execution result of the generated SQL query against the gold answer.

Input / output format

Input: Question text. For retrieval, compared against flattened table segments (tables split into 100-word chunks and appended with descriptions). For end-to-end QA, question concatenated with retrieved table segments.

Output: For retrieval: ranked list of table segments. For QA: textual answer or SQL query.

Scoring recipe

def score_retrieval(retrieved, gold):
    return 1.0 if gold in retrieved else 0.0

def score_reader_em(pred_answer, gold_answer):
    return 1.0 if pred_answer == gold_answer else 0.0

def score_parser_em(pred_sql, gold_answer, tables):
    exec_pred = execute_sql(pred_sql, tables)
    return 1.0 if exec_pred == gold_answer else 0.0

Common pitfalls

• Table splitting into 100-word chunks means models must handle multiple segments per table, which significantly impacts retrieval and QA performance.
• Parser EM relies on SQL execution results, so syntactically different but semantically equivalent SQL queries are treated as correct, unlike strict string matching.
• Decontextualized questions make retrieval artificially easy due to direct information overlap, while paraphrased questions require deeper semantic understanding.

Evidence (verbatim from paper)

We use the exact match accuracy (EM) for the evaluation metric. For the parser, EM is computed on the execution result of generated SQLs, as they can be expressed in a diverse form.

Citation

@misc{kweon2023openwikitable,
  title={Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table},
  author={Kweon et al. (2023)},
  year={2023},
  note={arXiv:2305.07288}
}

• arXiv: 2305.07288