qhjqhj00 quac-eval
QuAC : Question Answering in Context — Choi et al. (2018) (arXiv:1808.07036, 2018)
What this evaluates
Evaluates multi-turn, context-dependent question answering where models must track dialog history, resolve coreference, and handle open-ended follow-ups. It specifically probes a system's ability to manage asymmetric knowledge access and correctly identify unanswerable questions within an information-seeking dialogue.
Datasets
- QuAC — total 98100; splits: train (83500), dev (7300), test (7300)
Metrics
word-level F1(primary) — range: [0, 1]- Precision and recall computed on word overlap after removing stopwords. For no-answer questions, F1 is 1.0 if correct, 0.0 otherwise. The maximum F1 across all reference answers is taken, then averaged over all (n-1) subsets of references to fairly compare with human performance.
HEQ-Q— range: percent- Percentage of questions where the system's F1 exceeds or matches the human F1.
HEQ-D— range: percent- Percentage of dialogs where the system's F1 exceeds or matches human F1 for every question in the dialog.
dialog act accuracy— range: percent- Percentage of correctly predicted dialog acts compared to the majority annotation, with ties broken randomly.
Input / output format
Input: First k questions and k ground-truth answers in the dialog, supporting material (entity e, topic t, background b, and section text s), and the next question q_{k+1}.
Output: Answer span indices i, j in the section text s, dialog acts v, combined as the final answer prediction a_{k+1}.
Scoring recipe
def score_quac(preds, golds, human_f1s):
f1_scores = []
for pred, gold_list in zip(preds, golds):
if pred.span is None:
f1 = 1.0 if not gold_list else 0.0
else:
f1 = max(word_overlap_f1(pred.span, g) for g in gold_list)
f1_scores.append(f1)
# Average max F1 over all (n-1) subsets of references
avg_f1 = average_max_f1_over_subsets(f1_scores, golds)
# HEQ-Q: % questions where sys F1 >= human F1
heq_q = sum(1 for f1, h in zip(avg_f1, human_f1s) if f1 >= h) / len(golds)
# HEQ-D: % dialogs where sys F1 >= human F1 for all questions
heq_d = sum(all(f1 >= h for f1, h in zip(dialog_f1s, dialog_human_f1s)) for dialog in dialogs) / num_dialogs
# Dialog act accuracy
act_acc = sum(1 for p, g in zip(pred_acts, gold_acts) if p == g) / len(golds)
return avg_f1, heq_q, heq_d, act_acc
Common pitfalls
- F1 varies significantly with the number of reference annotations; must average the maximum F1 over all (n-1) subsets of references to compare fairly with human performance.
- Questions with human F1 lower than 40 are excluded (~10% of data) to remove noisy annotations, so evaluation should only run on the filtered set.
- No-answer questions yield an F1 of exactly 1.0 or 0.0, not a partial score, requiring strict binary matching for the 'no answer' prediction.
Evidence (verbatim from paper)
Our core evaluation metric, word-level F1, is implemented similarly to SQuAD (Rajpurkar et al., 2016): precision and recall are computed by considering the portion of words in the prediction and references that overlap after removing stopwords. For no answer questions, we give the system an F1 of one if it correctly predicts no answer and zero otherwise. ... we report the average of the maximum F1 computed from each n - 1 subset with respect to the heldout reference. Additionally, since averaged F1 can be misleading for questions with multiple valid answers, we introduce the human equivalence score (HEQ) ... HEQ measures the percentage of examples for which system F1 exceeds or matches human F1. We compute two variants: (1) the percentage of questions for which this is true (HEQ-Q), and (2) the percentage of dialogs for which this is true for every question in the dialog (HEQ-D).
Citation
@misc{choi2018quac,
title={QuAC : Question Answering in Context},
author={Choi et al. (2018)},
year={2018},
note={arXiv:1808.07036}
}
- arXiv: 1808.07036