Dstc11 Track3 Eval

dstc11-track3-eval

Adapting Text-based Dialogue State Tracker for Spoken Dialogues — Yoon et al. (2023) (arXiv:2308.15053, 2023)

What this evaluates

Evaluates a system's ability to track dialogue state in spoken conversations, specifically measuring robustness to ASR errors, disfluencies, and proper noun mismatches.

Datasets

• DSTC11 Track 3 — total ?; splits: tts-verbatim (-1), human-verbatim (-1), human-paraphrased (-1)

Metrics

• JGA (primary) — range: percent
  • Joint Goal Accuracy: the percentage of dialogue turns where all slot values are predicted exactly correctly.
• SER — range: percent
  • Slot Error Rate: the percentage of individual slots across all turns that are predicted incorrectly.

Input / output format

Input: Sequential dialogue turns provided as ASR transcripts or TTS-generated text.

Output: A set of slot-value pairs representing the current dialogue state.

Scoring recipe

def compute_jga(pred_states, gold_states):
    correct = sum(1 for p, g in zip(pred_states, gold_states) if p == g)
    return correct / len(gold_states)

def compute_ser(pred_states, gold_states):
    total, errors = 0, 0
    for p, g in zip(pred_states, gold_states):
        for slot in g:
            total += 1
            if slot not in p:
                errors += 1
    return errors / total

Common pitfalls

• Confusing evaluation splits: TTS-verbatim (machine-generated) vs human-verbatim/paraphrased (human speech) drastically changes ASR error profiles and results.
• Proper nouns dominate error rates due to ASR hallucinations and lack of external ontologies, often masking performance on common slots.

Evidence (verbatim from paper)

Table 5 is the official results of the test submission by the participants. A total of 6 teams submitted, and each team could submit up to 2 systems, so a total of 11 systems were submitted. We submitted a model that recorded 42.4 in the validation set. Finally, our model achieved third place, with JGA 40.2 for tts-verbatim in the challenge.

Citation

@misc{yoon2023adapting,
  title={Adapting Text-based Dialogue State Tracker for Spoken Dialogues},
  author={Yoon et al. (2023)},
  year={2023},
  note={arXiv:2308.15053}
}

• arXiv: 2308.15053