Speechmedbench Eval

speechmedbench-eval

SpeechMedAssist: Efficiently and Effectively Adapting Speech Language Models for Medical Consultation — Sirry Chen et al. (2026) (arXiv:2601.04638, 2026)

What this evaluates

Evaluates speech language models on medical consultation tasks, covering single-turn medical knowledge Q&A, multi-turn diagnostic conversations, real-world clinical robustness, and speech output quality.

Datasets

• SpeechMedBench — total ?; splits: test (-1); repo https://github.com/SirryChen/SpeechMedAssist
• CMB — total ?; splits: test (-1)
• CME — total ?; splits: test (-1)
• MedDG — total ?; splits: test (-1)
• AIHospital — total ?; splits: test (-1)
• MedSafetyBench — total ?; splits: test (-1)
• Wild — total 20; splits: test (20)

Metrics

• CMB (primary) — range: percent
  • Accuracy on medical multiple-choice questions. Evaluated in text-only form.
• CME (primary) — range: percent
  • Accuracy on medical multiple-choice questions. Evaluated in text-only form.
• Ency — range: percent
  • Score on medical encyclopedia Q&A pairs sampled from Huatuo2-pretrain.
• Safety — range: [1, 5]
  • MedSafetyBench score ranging from 1 to 5. Lower scores indicate better medical safety.
• MedDG (primary) — range: [0, 100]
  • LLM-judged score (Qwen2.5-72B) on multi-turn consultations simulated with MedDG patient profiles across six perspectives.
• AIHospital (primary) — range: [0, 100]
  • LLM-judged score (Qwen2.5-72B) on multi-turn consultations using real patient cases from AIHospital.
• Vote — range: count
  • Total votes from five medical professionals on 20 real clinical recordings, selecting the most clinically appropriate response.
• UTMOS — range: [1, 5]
  • Mean Opinion Score predicted by a MOS prediction model to measure speech naturalness.
• ASR-CER — range: percent
  • Character Error Rate computed by transcribing generated speech with an ASR model and comparing against target text.
• Latency — range: ms
  • Time in milliseconds from the start of speech input to the generation of the first speech chunk.

Input / output format

Input: Speech audio input (single-turn or multi-turn) with optional text context. For CMB and CME, plain text prompts are used.

Output: Text response followed by synthesized speech audio.

Scoring recipe

def score(predictions, gold):
    scores = {}
    scores['CMB'] = accuracy(predictions['CMB'], gold['CMB'])
    scores['CME'] = accuracy(predictions['CME'], gold['CME'])
    scores['Ency'] = llm_judge_score(predictions['Ency'], gold['Ency'])
    scores['Safety'] = mean([safety_score(p) for p in predictions['Safety']])
    scores['MedDG'] = llm_judge_score(predictions['MedDG'], gold['MedDG'])
    scores['AIHospital'] = llm_judge_score(predictions['AIHospital'], gold['AIHospital'])
    scores['Vote'] = sum(1 for v in predictions['Wild'] if v == gold['Wild'])
    scores['UTMOS'] = utmos_model(predictions['Speech'])
    scores['ASR-CER'] = cer(predictions['Speech'], gold['Text'])
    scores['Latency'] = time_to_first_chunk(predictions['Speech'])
    return scores

Common pitfalls

• CMB and CME are evaluated in text-only form, not via speech input, despite the model being a SpeechLM.
• The Safety metric is inverted: lower scores indicate better medical safety (1-5 scale).
• Multi-turn conversation scores (MedDG, AIHospital) are highly sensitive to the LLM judge's prompt and inherent model biases.
• Wild evaluation relies on a small sample (20 recordings) and human voting, which may not generalize to broader clinical scenarios.

Evidence (verbatim from paper)

To evaluate our model and compare it with baselines, we construct SpeechMedBench and evaluate mainly four dimensions: medical knowledge, diagnostic capability, robustness, and speech quality. ... we use evaluation sets of two medical multiple-choice datasets, CMB (Wang et al., [2024]) and CME (Liu et al., [2023]), along with medical encyclopedia Q&A pairs randomly sampled from the Huatuo2-pretrain dataset (referred to as Ency), which cover a wide range of medical terminology without overlapping with the training data. We also adopt MedSafetyBench (referred to as Safety) (Han et al., [2024]) to evaluate the medical safety of models, with scores ranging from 1 to 5. ... Finally, a chief examiner powered by Qwen2.5-72B acting as an LLM-based judge (Zheng et al., [2023]) evaluates dialogues from six perspectives, as detailed in Appendix[I]. ... We evaluate speech response quality from three aspects: (1) UTMOS measures speech naturalness using a MOS prediction model (Saeki et al., [2022]); (2) ASR-CER evaluates text–speech consistency by transcribing the generated speech with an ASR model and computing the character error rate against the target text; and (3) Latency is the time from th

Citation

@misc{chen2026speechmedassist,
  title={SpeechMedAssist: Efficiently and Effectively Adapting Speech Language Models for Medical Consultation},
  author={Sirry Chen et al. (2026)},
  year={2026},
  note={arXiv:2601.04638}
}

• arXiv: 2601.04638