Structured Data QA Eval

structured-data-qa-eval

Self-Correction Distillation for Structured Data Question Answering — Zhu et al. (2025) (arXiv:2511.07998, 2025)

What this evaluates

Evaluates the ability of LLMs to generate executable structured queries (SQL, SPARQL) from natural language questions over tables, knowledge graphs, and temporal knowledge graphs. It specifically probes the model's capacity for iterative self-correction when initial query generation fails, measuring both raw generation accuracy and error-resilience during multi-round inference.

Datasets

• WikiSQL — total ?; splits: train (56351), dev (3000), test (15878)
• WTQ — total ?; splits: train (11322), dev (2830), test (4344)
• MetaQA — total ?; splits: train (30000), dev (6000), test (39093)
• WebQSP — total ?; splits: train (2788), dev (308), test (1639)
• CronQuestions — total ?; splits: train (25000), dev (4000), test (30000)
• TabFact — total ?; splits: test (-1)

Metrics

• Denotation accuracy (primary) — range: [0, 1]
  • Checks if the execution result of the generated query exactly matches the execution result of the ground truth query on the target database.
• Hit@1 — range: [0, 1]
  • Checks if the ground truth answer is present in the top-1 generated or retrieved result.
• Accuracy — range: [0, 1]
  • Standard exact-match accuracy for the TabFact dataset.

Input / output format

Input: Natural language question paired with retrieved structured data context (e.g., table rows/columns, KB triples, or temporal facts). The system uses a dense text encoder to retrieve up to 15 relevant demonstrations/snippets.

Output: A single executable structured query string (e.g., SQL for tables, SPARQL/KB query for KGs). During EPM inference, the model may output intermediate correction steps before the final query.

Scoring recipe

def score_denotation_accuracy(generated_query, gold_query, db_context):
    gen_result = execute_query(generated_query, db_context)
    gold_result = execute_query(gold_query, db_context)
    return 1.0 if set(gen_result) == set(gold_result) else 0.0

def score_hit_at_1(generated_answers, gold_answer):
    return 1.0 if gold_answer in generated_answers[:1] else 0.0

Common pitfalls

• WTQ lacks an official validation set; the authors manually split the raw training set 80-20 for development, which differs from standard splits.
• Large datasets (MetaQA, CronQuestions) are uniformly sampled to balance class/hop distribution and reduce OpenAI costs, altering the original dataset scale.
• Results must be reported separately for 'w/ EPM' (iterative multi-round correction) and 'w/o EPM' (single-pass), as performance differs significantly.

Evidence (verbatim from paper)

Table 3: Denotation accuracy of TableQA, Hit@1 of KG QA and temporal KG QA. w/ denotes “inference w/ EPM”, and w/o denotes “inference w/o EPM”.

Citation

@misc{zhu2025selfcorrectiondistillation,
  title={Self-Correction Distillation for Structured Data Question Answering},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2511.07998}
}

• arXiv: 2511.07998