Contextdialog Eval

contextdialog-eval

Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models — Kim et al. (2025) (arXiv:2502.19759, 2025)

What this evaluates

Evaluates conversational context recall and utilization in voice interaction models, specifically measuring how well they remember and respond to past user and system utterances in multi-turn dialogues. It also probes the robustness of retrieval-augmented generation (RAG) when applied to speech-based models.

Datasets

• ContextDialog — total ?; splits: test (-1); repo https://github.com/dvlab-research/Lyra

Metrics

• GPT Score (primary) — range: [0, 5]
  • A 5-point scale score generated by an LLM-as-a-judge (gpt-4o-mini) that rates how well a model's generated response contains the ground truth information relevant to a given question about past dialog context. Higher scores indicate better recall and utilization.
• WER — range: percent
  • Word Error Rate calculated between the model's intermediate text response and the Whisper-transcribed spoken response, used to disentangle recall ability from speech synthesis degradation.

Input / output format

Input: Multi-turn conversational context (speech audio) followed by a question/query requiring recall of past utterances.

Output: Spoken response audio (and optionally an intermediate text response). For evaluation, speech is transcribed to text via Whisper-large-v3 before scoring.

Scoring recipe

def compute_gpt_score(model_speech, question, ground_truth):
    text_resp = whisper_transcribe(model_speech)
    prompt = f'Q: {question}\nGT: {ground_truth}\nResp: {text_resp}\nScore 1-5 on recall.'
    return gpt4o_mini_evaluate(prompt)

def compute_wer(text_resp, model_speech):
    transcribed = whisper_transcribe(model_speech)
    return wer_score(text_resp, transcribed)

Common pitfalls

• Confusing speech synthesis errors with poor context recall; the protocol disentangles these by evaluating intermediate text responses alongside transcribed speech.
• Assuming RAG improves performance; the evaluation shows retrieval errors and prompt length often degrade voice model responses.
• Ignoring modality bias; models consistently score higher on recalling their own past utterances than user utterances due to generation mechanisms.

Evidence (verbatim from paper)

In all experiments, we evaluate each model's spoken response using the LLM-as-a-judge approach (Zheng et al., 2023), following previous works (Chen et al., 2024c; Zeng et al., 2025). This setting is denoted as $S ightarrow T$ , $\underline{S}$ , where $S$ refers to speech data and $T$ to text data. The bold and underlined modality symbol indicates the evaluation target in each configuration. We employ gpt-4o-mini for evaluation, referred to as the GPT Score in this paper, using a five-point scale where higher scores indicate better performance. We design prompts to assess recall by measuring how well the generated responses contain the ground truth information relevant to the given question, as detailed in Appendix A.4. Since gpt-4o-mini is tailored to text inputs, we first convert the spoken responses into text using whisper-large-v3 (Radford et al., 2023).

Citation

@misc{kim2025doesyourvoiceassistantremember,
  title={Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models},
  author={Kim et al. (2025)},
  year={2025},
  note={arXiv:2502.19759}
}

• arXiv: 2502.19759