Topiocqa Eval

topiocqa-eval

TopiOCQA: Open-domain Conversational Question Answering with Topic Switching — Adlakha et al. (2021) (arXiv:2110.00768, 2021)

What this evaluates

Evaluates open-domain conversational question answering with topic switching, requiring models to maintain context across multiple turns and dynamically retrieve relevant documents to answer evolving questions.

Datasets

• TOPIOCQA — total ?; splits: dev (-1), test (-1)

Metrics

• exact match (EM) — range: [0, 1]
  • Binary score of 1 if the predicted answer exactly matches any of the gold answers, else 0. Averaged across all instances and reference sets.
• F1 (primary) — range: [0, 1]
  • Harmonic mean of token-level precision and recall between the predicted answer and gold answers. Averaged across all instances and reference sets.

Input / output format

Input: Conversation history consisting of previous question-answer pairs {q_1, a_1, ..., q_{i-1}, a_{i-1}} followed by the current question q_i. Optionally augmented with retrieved document passages from a corpus C.

Output: A free-form text answer a_i.

Scoring recipe

def compute_multi_ref_score(predictions, gold_answers_list):
    total_scores = []
    for pred, gold_list in zip(predictions, gold_answers_list):
        n = len(gold_list)
        instance_scores = []
        for i in range(n):
            # Create reference set excluding the i-th answer
            ref_set = [gold_list[j] for j in range(n) if j != i]
            # Compute metric against each reference in the set
            ref_scores = [compute_em_or_f1(pred, ref) for ref in ref_set]
            instance_scores.append(sum(ref_scores) / len(ref_scores))
        # Average over the n reference sets
        total_scores.append(sum(instance_scores) / n)
    return sum(total_scores) / len(total_scores)

Common pitfalls

• Failing to properly average scores across multiple gold annotations per turn, which can artificially inflate or deflate performance.
• Ignoring conversation history length constraints when using the ALLHISTORY representation, leading to truncated context and degraded retrieval.
• Assuming human performance is an absolute upper bound, whereas systems can surpass humans by retrieving better documents than those used by annotators.

Evidence (verbatim from paper)

Following Choi et al. (2018) and Reddy et al. (2019), we use exact match (EM) and F1 as evaluation metrics for TOPIOCQA. To compute human and system performance in the presence of multiple gold annotations, we follow the evaluation process similar to Choi et al. (2018) and Reddy et al. (2019). Given n human answers, human performance on the task is determined by considering each answer as prediction and other human answers as the reference set. This results in n scores, which are averaged to give the final human performance score. The system prediction is also compared with n distinct reference sets, each containing n-1 human answers, and then averaged.

Citation

@misc{adlakha2021topiocqa,
  title={TopiOCQA: Open-domain Conversational Question Answering with Topic Switching},
  author={Adlakha et al. (2021)},
  year={2021},
  note={arXiv:2110.00768}
}

• arXiv: 2110.00768