Dstc10 Spoken Eval

dstc10-spoken-eval

"How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations — Kim et al. (2021) (arXiv:2109.13489, 2021)

What this evaluates

Evaluates task-oriented dialogue systems on spoken conversations to measure robustness against ASR errors and disfluencies. It probes multi-domain dialogue state tracking, knowledge-seeking turn detection, knowledge selection, and response generation capabilities under realistic speech conditions.

Datasets

• DSTC10 — total 263; splits: valid (263); repo https://github.com/alexa/alexa-with-dstc9-track1-dataset
• DSTC9 — total 4181; splits: test (4181)
• MultiWOZ 2.1 — total ?; splits: test (-1)

Metrics

• Joint Goal Accuracy (primary) — range: [0, 1]
  • Percentage of dialogue turns where all slot values are predicted exactly correctly.
• Slot F1 — range: [0, 1]
  • F1 score computed separately for value predictions and none predictions, reported independently or averaged.
• Detection F1 — range: [0, 1]
  • F1 score for binary knowledge-seeking turn detection.
• MRR@5 — range: [0, 1]
  • Mean Reciprocal Rank of the correct knowledge item within the top 5 retrieved candidates.
• BLEU-4 — range: [0, 1]
  • 4-gram BLEU score comparing generated responses to reference responses.
• Human Accuracy — range: [1, 5]
  • Crowd-sourced score on a 1-5 scale assessing how accurate the system output is relative to reference knowledge.

Input / output format

Input: Dialogue context and target user turn provided as ASR transcripts (1-best or n-best hypotheses), along with domain and knowledge information for response generation.

Output: For Task 1: predicted slot values and 'none' flags per slot. For Task 2: binary turn detection label, ranked list of knowledge items, and generated response text.

Scoring recipe

def score_dst(pred_slots, gold_slots):
    joint_acc = sum(1 for p, g in zip(pred_slots, gold_slots) if p == g) / len(gold_slots)
    val_f1 = f1_score(gold_values, pred_values)
    none_f1 = f1_score(gold_none, pred_none)
    return joint_acc, val_f1, none_f1

def score_detection(pred_det, gold_det):
    return f1_score(gold_det, pred_det)

def score_selection(pred_ranks, gold_idx):
    mrr = mean(1.0 / r for r in pred_ranks if r <= 5)
    rec1 = mean(1 if r == 1 else 0 for r in pred_ranks)
    return mrr, rec1

def score_generation(preds, refs):
    return compute_bleu(refs, preds, n=4), compute_rouge(refs, preds)

def score_human(preds, refs):
    scores = [crowd_worker_score(p, r) for p, r in zip(preds, refs)]
    return mean(scores)  # 1-5 scale

Common pitfalls

• Comparing spoken DSTC10 directly with written MultiWOZ/DSTC9 without accounting for modality/locale differences, which causes significant performance drops independent of ASR errors.
• Evaluating only on the top-1 ASR result ignores the paper's finding that ensemble/n-best hypothesis aggregation is necessary for robust spoken dialogue modeling.
• Focusing solely on joint goal accuracy while overlooking slot-level precision/recall trade-offs, which masks false negative issues in value predictions.

Evidence (verbatim from paper)

It achieves extremely low performance on the spoken data in joint goal accuracy and also a significantly worse score in slot-level accuracy compared to that on MultiWOZ.

Citation

@misc{kim2021howrobust,
  title={"How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations},
  author={Kim et al. (2021)},
  year={2021},
  note={arXiv:2109.13489}
}

• arXiv: 2109.13489