Hybridqa Eval

hybridqa-eval

HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data — Wenhu Chen et al. (arXiv:2004.07347, 2020)

What this evaluates

Multi-hop question answering that requires integrating information from both tabular and textual sources. It probes a model's ability to perform cross-modal reasoning and extract precise answers from heterogeneous data.

Datasets

• HybridQA — total ?; splits: test (500), dev (-1); repo https://github.com/wenhuchen/HybridQA

Metrics

• exact match (EM) (primary) — range: [0, 1]
  • 1 if the predicted answer exactly matches the ground-truth answer string, 0 otherwise.
• F1 — range: [0, 1]
  • Token-level F1 score measuring the average overlap between the prediction and ground-truth answers.

Input / output format

Input: A natural language question, a retrieved table (or specific cell), and a retrieved text passage.

Output: A text span extracted from the table cell or passage that answers the question.

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores = [1.0 if pred == gold else 0.0 for pred, gold in zip(predictions, golds)]
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        p_tokens = set(pred.lower().split())
        g_tokens = set(gold.lower().split())
        if not p_tokens or not g_tokens:
            f1_scores.append(0.0)
            continue
        intersection = len(p_tokens & g_tokens)
        precision = intersection / len(p_tokens)
        recall = intersection / len(g_tokens)
        f1_scores.append(2 * precision * recall / (precision + recall))
    return {'EM': sum(em_scores) / len(em_scores), 'F1': sum(f1_scores) / len(f1_scores)}

Common pitfalls

• Models using only table-only or passage-only inputs achieve <20% accuracy, so cross-modal integration is mandatory.
• Errors cascade across retrieval, ranking, hop, and reading comprehension stages, making step-wise error analysis critical for debugging.
• The retrieval threshold τ trades off recall vs. precision, but the overall model performance remains relatively stable across different τ values.

Evidence (verbatim from paper)

Following previous work, we use exact match (EM) and F1 as two evaluation metrics. F1 metric measures the average overlap between the prediction and ground-truth answers.

Citation

@misc{chen2020hybridqa,
  title={HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data},
  author={Wenhu Chen et al.},
  year={2020},
  note={arXiv:2004.07347}
}

• arXiv: 2004.07347