Tempqa Wd Eval

tempqa-wd-eval

A Benchmark for Generalizable and Interpretable Temporal Question Answering over Knowledge Bases — Neelam et al. (2022) (arXiv:2201.05793, 2022)

What this evaluates

Probes a system's ability to perform temporal question answering over knowledge bases by generating correct answers or SPARQL queries. It specifically evaluates generalization across different knowledge bases (Wikidata vs. Freebase) and interpretability through fine-grained intermediate annotations like entity/relation linking and λ-expressions.

Datasets

• TempQA-WD — total 510; splits: dev (-1), test (-1); repo https://github.com/IBM/tempqa-wd

Metrics

• macro precision — range: [0, 1]
  • Standard macro-averaged precision computed over gold vs. system-generated answers per question.
• macro recall — range: [0, 1]
  • Standard macro-averaged recall computed over gold vs. system-generated answers per question.
• F1 (primary) — range: [0, 1]
  • Harmonic mean of macro precision and macro recall, computed as 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Natural language temporal questions targeting entities/relations in a knowledge base (Wikidata or Freebase).

Output: System-generated answers or SPARQL queries for each question.

Scoring recipe

precisions, recalls = [], []
for q in questions:
    gold = set(gold_answers[q])
    pred = set(system_answers[q])
    if len(gold) == 0: continue
    p = len(gold & pred) / len(pred) if len(pred) > 0 else 0
    r = len(gold & pred) / len(gold)
    precisions.append(p)
    recalls.append(r)
macro_p = sum(precisions) / len(precisions)
macro_r = sum(recalls) / len(recalls)
f1 = 2 * macro_p * macro_r / (macro_p + macro_r) if (macro_p + macro_r) > 0 else 0

Common pitfalls

• The dataset is heavily skewed toward simple questions (471 simple vs. 39 complex), which can mask poor performance on complex temporal reasoning.
• Performance bottlenecks often lie in intermediate steps like entity and relation linking on Wikidata rather than the core temporal reasoning module.

Evidence (verbatim from paper)

We use GERBIL (Usbeck et al., 2019) to compute performance metrics from the pairs of gold answers and system generated answers from the pipeline. We use standard performance metrics typically used for KBQA systems, namely macro precision, macro recall and F1.

Citation

@misc{neelam2022tempqa,
  title={A Benchmark for Generalizable and Interpretable Temporal Question Answering over Knowledge Bases},
  author={Neelam et al. (2022)},
  year={2022},
  note={arXiv:2201.05793}
}

• arXiv: 2201.05793