Duplex Dialogue Eval

duplex-dialogue-eval

A Full-duplex Speech Dialogue Scheme Based On Large Language Models — Peng Wang et al. (2024) (arXiv:2405.19487, 2024)

What this evaluates

Evaluates a dialogue system's ability to manage full-duplex speech interactions, specifically focusing on the timing and appropriateness of machine-to-user interruptions and user-to-machine interruptions, alongside system response latency.

Datasets

• duplex-dialogue-3k — total 3000; splits: test (3000)

Metrics

• FTED (primary) — range: other
  • First Token Emission Delay measures the time between the end of the user's speech and the machine's first output token. It captures the total pipeline latency including ASR, LLM, and TTS components.
• PIR_mid — range: percent
  • Proper Interruption Rate for mid-sentence interruptions. It calculates the proportion of machine interruptions occurring mid-sentence that are deemed appropriate or reasonable by human or GPT-4 evaluation.
• PRR_mid — range: percent
  • Proper Response Rate for mid-sentence interruptions. It measures the proportion of reasonable or appropriate textual responses generated by the model immediately after a mid-sentence interruption.
• Precision — range: percent
  • A composite interruption precision metric defined as PIR_mid * ir_mid + ir_end, where ir_mid and ir_end are the proportions of mid-sentence and sentence-completion interruptions respectively.
• Recall — range: percent
  • Approximated as 1 - MIR, where MIR is the Missed Interruption Rate (the proportion of cases where the model fails to interrupt the user when it should).
• PRRavg — range: percent
  • Average Proper Response Rate across four user-to-machine interruption categories: noise, denial, affirmation, and shifting the topic.

Input / output format

Input: Streaming text transcripts of multi-turn dialogues. For machine interruption tests, the final user utterance is fed token-by-token. For user interruption tests, the final turn contains a specific interruption type (noise, denial, affirmation, or topic shift).

Output: Textual response and control tokens for FSM state transitions (SPEAK/LISTEN). Evaluation captures the exact timestamp of the first output token and judges the appropriateness of the interruption and response.

Scoring recipe

fted = timestamp(first_output_token) - timestamp(user_speech_end)
ir_mid = count(mid_sentence_interruptions) / total_cases
ir_end = count(sentence_completion_interruptions) / total_cases
MIR = count(no_interruption) / total_cases
PIR_mid = count(appropriate_mid_interruptions) / count(mid_sentence_interruptions)
PRR_mid = count(appropriate_responses_after_mid_interruption) / count(mid_sentence_interruptions)
precision = PIR_mid * ir_mid + ir_end
recall = 1 - MIR
PRRavg = mean(PRR_noise, PRR_denial, PRR_affirm, PRR_shift)

Common pitfalls

• The Precision and Recall formulas are non-standard and specific to this paper's interruption framework; do not use standard binary classification formulas.
• ASR VAD latency is explicitly excluded from the LLM's evaluation scope; FTED measures the full system pipeline delay, not just LLM inference time.
• Metrics are strictly split by interruption direction (machine interrupts user vs. user interrupts machine) and timing (mid-sentence vs. end-sentence); mixing these categories will invalidate results.

Evidence (verbatim from paper)

We formulate several assessment metrics to test dialogue performance. We use a metric proper interruption rate (PIR) to measure the accuracy of the interruption timing, and the metric proper response rate (PRR) to indicate the proportion of reasonable responses generated by the model after an interruption. For the overall system latency, metric first token emission delay (FTED) is considered: the time between the end of the user's speech and the machine's first output.

Citation

@misc{wang2024fullduplex,
  title={A Full-duplex Speech Dialogue Scheme Based On Large Language Models},
  author={Peng Wang et al. (2024)},
  year={2024},
  note={arXiv:2405.19487}
}

• arXiv: 2405.19487