Spoken Coqa Eval

spoken-coqa-eval

Towards Data Distillation for End-to-end Spoken Conversational Question Answering — Chenyu You et al. (arXiv:2010.08923, 2020)

What this evaluates

Evaluates conversational question answering models on both clean text and noisy ASR transcripts, measuring their ability to maintain performance under speech recognition errors and leverage data distillation techniques.

Datasets

• CoQA — total ?; splits: train (-1), dev (-1)
• Spoken-CoQA — total ?; splits: train (-1), test (-1)

Metrics

• EM — range: [0, 1]
  • Exact Match (EM) is 1 if the predicted answer exactly matches any of the gold answers, else 0.
• F1 (primary) — range: [0, 1]
  • Token-level F1 score averaged over all gold answers for each question, then averaged across the dataset.

Input / output format

Input: Context document, multi-turn dialogue history, and a question. Inputs are provided either as clean text or as ASR-generated transcripts.

Output: Predicted answer span or text string.

Scoring recipe

def compute_metrics(predictions, gold_answers):
    em_scores = []
    f1_scores = []
    for pred, golds in zip(predictions, gold_answers):
        em_scores.append(1.0 if pred in golds else 0.0)
        f1s = [token_f1(pred, g) for g in golds]
        f1_scores.append(max(f1s))
    return {'EM': sum(em_scores) / len(em_scores), 'F1': sum(f1_scores) / len(f1_scores)}

Common pitfalls

• Performance drops significantly when models are trained on ASR transcripts instead of clean text due to speech recognition errors.
• Local training environments may yield different results compared to the official CoQA leaderboard.
• Tokenization differences (SpaCy vs. BPE) require embedding averaging for fair comparison across baselines.

Evidence (verbatim from paper)

To train the teacher-student pairs simultaneously, we first train baselines on the CoQA training set and then compare the performances of testing baselines on CoQA dev set and Spoken-CoQA dev set. Finally, we train the baselines on the Spoken-CoQA training set and evaluate the baselines on the CoQA dev set and Spoken-CoQA test set. For evaluation, we use Exact Match (EM) and F1 score to compare the model performance on the test set.

Citation

@misc{you2020towards,
  title={Towards Data Distillation for End-to-end Spoken Conversational Question Answering},
  author={Chenyu You et al.},
  year={2020},
  note={arXiv:2010.08923}
}

• arXiv: 2010.08923