Ott QA Eval

ott-qa-eval

Open Question Answering over Tables and Text — Chen et al. (2020) (arXiv:2010.10439, 2020)

What this evaluates

This benchmark evaluates a model's ability to perform open-domain question answering by retrieving and fusing evidence from both tabular and textual sources. It specifically probes multi-hop reasoning capabilities where answers require bridging information across separate table segments and text passages.

Datasets

• OTT-QA — total ?; splits: dev (-1), test (-1); repo https://github.com/wenhuchen/OTT-QA

Metrics

• EM (primary) — range: percent
  • Exact match accuracy; the predicted answer string must exactly match the ground truth answer string.
• F1 — range: percent
  • Token-level F1 score measuring the overlap between predicted and ground truth answer tokens.

Input / output format

Input: A natural language question and a set of retrieved evidence blocks (table segments and text passages), typically truncated to 4096 subword tokens for the reader model.

Output: A single text string representing the predicted answer.

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores = [1.0 if pred.strip() == gold.strip() else 0.0 for pred, gold in zip(predictions, golds)]
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        p_tokens = set(pred.lower().split())
        g_tokens = set(gold.lower().split())
        if not p_tokens or not g_tokens:
            f1_scores.append(0.0)
            continue
        intersection = len(p_tokens & g_tokens)
        precision = intersection / len(p_tokens)
        recall = intersection / len(g_tokens)
        f1_scores.append(2 * precision * recall / (precision + recall))
    return sum(em_scores) / len(em_scores) * 100, sum(f1_scores) / len(f1_scores) * 100

Common pitfalls

• The evaluation jointly measures retrieval and reading performance; poor entity linking or block retrieval drastically lowers the final EM/F1, masking reader capabilities.
• Using predicted hyperlinks instead of oracle links causes a significant performance drop (~7% EM), highlighting that link prediction is a major bottleneck in this task.
• Single-hop retrieval often fails to capture bridging evidence, so models must use iterative or fusion retrieval strategies to achieve competitive scores.

Evidence (verbatim from paper)

By combining the two strategies, the final EM score can reach 28%, with an 18% absolute improvement, which is greater than the sum of individual improvements.

Citation

@misc{chen2020ottqa,
  title={Open Question Answering over Tables and Text},
  author={Chen et al. (2020)},
  year={2020},
  note={arXiv:2010.10439}
}

• arXiv: 2010.10439