Drvoice Eval

drvoice-eval

DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations — Tan et al. (2025) (arXiv:2506.09349, 2025)

What this evaluates

Evaluates a speech-text voice conversation model's capabilities in speech-to-text understanding, speech-to-speech generation, and overall speech quality. It probes modality alignment, reasoning, open-ended QA, and instruction following across multiple audio benchmarks.

Datasets

• OpenAudioBench — total ?; splits: test (-1); HF baichuan-inc/OpenAudioBench
• VoiceBench — total ?; splits: test (-1)
• UltraEval-Audio — total ?; splits: test (-1); repo https://github.com/OpenBMB/UltraEval-Audio
• Big Bench Audio — total ?; splits: test (-1); HF ArtificialAnalysis/big_bench_audio

Metrics

• G-Eval — range: [1, 5]
  • LLM-based scoring using G-Eval (Liu et al., 2023) to evaluate open-ended QA responses against gold references.
• Accuracy (primary) — range: [0, 1]
  • Exact match or token-level F1 score between predicted and gold answers for closed-ended benchmarks.
• Refusal Rate — range: [0, 1]
  • Percentage of model responses that explicitly refuse to answer adversarial prompts.
• ASR-WER — range: percent
  • Word Error Rate computed by transcribing generated speech with Whisper-v3-large and comparing against ground truth text.
• UTMOS — range: [1, 5]
  • Predicted score from the UTMOS model (Saeki et al., 2022) assessing overall naturalness and quality of generated speech.

Input / output format

Input: Audio waveform (tokenized at 5Hz, 12.5Hz, or 25Hz) concatenated with text prompts or conversation history.

Output: Text tokens (for S2T) or parallel speech tokens followed by text tokens (for S2S).

Scoring recipe

def evaluate(predictions, gold, task_type):
    if task_type == 'open_qa':
        return g_eval_score(predictions, gold)
    elif task_type == 'closed_qa':
        return exact_match_or_f1(predictions, gold)
    elif task_type == 'adversarial':
        return count_refusals(predictions) / len(predictions)
    elif task_type == 'speech_gen':
        transcribed = whisper_v3_large.predict(predictions)
        wer = compute_wer(transcribed, gold)
        utmos = utmos_model.predict(predictions)
        return {'ASR-WER': wer, 'UTMOS': utmos}

Common pitfalls

• Baseline results for GLM4-Voice are cited from external papers rather than re-run, so direct numerical comparison may not be strictly fair.
• ASR-WER scores are highly dependent on the Whisper-v3-large transcription model; using a different ASR backend will yield different WER values.
• G-Eval scores are LLM-dependent and can vary based on the scorer model version and prompt template, unlike deterministic accuracy metrics.

Evidence (verbatim from paper)

Evaluations adhere to the established protocols for each respective benchmark. Specifically, for the open-ended QA tasks on AlpacaEval and CommonEval, G-Eval (Liu et al., 2023) is used for scoring. For AdvBench, the Refusal Rate is reported, while performance on all other benchmarks is assessed with Accuracy. The generated speech is transcribed using Whisper-v3-large model (Radford et al., 2022), then WER (denoted by ASR-WER) of transcripts is computed against the generated text to assess the alignment between generated speech and text. UTMOS (Saeki et al., 2022) is used to evaluate the overall speech quality, following Zeng et al. (2025).

Citation

@misc{tan2025drvoice,
  title={DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations},
  author={Tan et al. (2025)},
  year={2025},
  note={arXiv:2506.09349}
}

• arXiv: 2506.09349