Coqa Eval

coqa-eval

CoQA: A Conversational Question Answering Challenge — Reddy et al. (2018) (arXiv:1808.07042, 2018)

What this evaluates

Evaluates a model's ability to answer free-form questions in a multi-turn conversational setting. It probes coreference resolution, pragmatic reasoning, and the capacity to maintain and leverage dialogue history over a given context passage.

Datasets

• CoQA — total ?; splits: dev (-1), test (-1)

Metrics

• macro-average F1 score of word overlap (primary) — range: [0, 1]
  • Word overlap F1 between the predicted answer and each of the n=4 gold answers. Articles (a, an, the) and punctuation are stripped before comparison. For each question, the maximum F1 across the n gold answers is taken, and these maximums are averaged across all questions.

Input / output format

Input: A context passage, the conversation history (previous question-answer pairs), and the current question.

Output: A free-form text answer (and optionally a rationale span from the passage).

Scoring recipe

def normalize(text):
    return re.sub(r'\b(a|an|the)\b', '', text).replace('.', '').replace(',', '')
def compute_metric(predictions, golds):
    f1_scores = []
    for pred, gold_list in zip(predictions, golds):
        pred_n = normalize(pred)
        max_f1 = 0
        for g in gold_list:
            max_f1 = max(max_f1, f1_score(pred_n, normalize(g)))
        f1_scores.append(max_f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Forgetting to strip articles ('a', 'an', 'the') and punctuation during answer normalization.
• Averaging F1 against only the first gold answer instead of taking the maximum F1 across all 4 gold answers per question.
• Treating the task as strict span extraction; the dataset requires free-form generation and the metric measures lexical overlap, not semantic equivalence.

Evidence (verbatim from paper)

Following SQuAD, we use macro-average F1 score of word overlap as our main evaluation metric. We use the gold answers of history to predict the next answer. In SQuAD, for computing a model's performance, each individual prediction is compared against $n$ human answers resulting in $n$ F1 scores, the maximum of which is chosen as the prediction's F1. For each question, we average out F1 across these $n$ sets, both for humans and models. In our final evaluation, we use $n = 4$ human answers for every question (the original answer and 3 additionally collected answers). The articles $a$, $an$ and the and punctuations are excluded in evaluation.

Citation

@misc{reddy2018coqa,
  title={CoQA: A Conversational Question Answering Challenge},
  author={Reddy et al. (2018)},
  year={2018},
  note={arXiv:1808.07042}
}

• arXiv: 1808.07042