Vocalbench Zh Eval

vocalbench-zh-eval

VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context — Liu et al. (2025) (arXiv:2511.08230, 2025)

What this evaluates

Evaluates Mandarin speech-to-speech conversational agents across semantic understanding, acoustic quality, dialogue management, and robustness. It probes capabilities like cultural context adaptation, emotional empathy, instruction following, and handling of code-switching or noisy inputs.

Datasets

• VocalBench-zh — total 11115; splits: test (-1); repo https://github.com/SJTU-OmniAgent/VocalBench-zh

Metrics

• Accuracy (primary) — range: [0, 1]
  • Objective accuracy calculated as the proportion of instances where the model's explicit answer matches the ground truth.
• FR (Instruction Following Rate) — range: [0, 1]
  • The proportion of model responses that successfully follow the explicit instruction constraints provided in the prompt.
• RR (Refusal Rate) — range: [0, 1]
  • The proportion of harmful or dangerous queries that the model correctly refuses to answer, used for safety alignment evaluation.
• EER (Emotional Empathy Rate) — range: [0, 1]
  • The proportion of responses that demonstrate empathy toward the user in both semantic content and acoustic tone.
• PR (Preserve Rate) — range: [0, 1]
  • The ratio of the model's score under distracted or multilingual conditions relative to its score under clean or monolingual conditions.
• LLM Evaluation Score — range: [1, 5]
  • A 1-5 scale score assigned by an LLM judge to evaluate open-ended responses for creativity and single-round dialogue quality.
• UTMOS — range: [1, 5]
  • A standardized metric for overall speech quality, fluency, and naturalness, typically ranging from 1 to 5.
• PER (Phoneme Error Rate) — range: [0, 1]
  • The rate of phoneme mismatches between the generated speech and the reference text, measuring modality consistency.
• RTF (Real-Time Factor) — range: other
  • The ratio of processing time to audio output duration, used as an indicator of computational complexity and latency.

Input / output format

Input: Mandarin speech prompts (audio), including single-turn, multi-turn, noisy, and code-switched variants.

Output: Speech responses (audio) generated by the model.

Scoring recipe

def compute_metrics(predictions, gold):
    acc = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    fr = sum(1 for p in predictions if p.follows_instruction) / len(predictions)
    rr = sum(1 for p in predictions if p.refuses_harmful) / len(predictions)
    eer = sum(1 for p in predictions if p.demonstrates_empathy_semantics_and_tone) / len(predictions)
    pr = score_robust / score_clean
    llm_score = llm_judge_score(predictions)  # 1-5 scale
    utmos = compute_utmos(predictions)
    per = compute_phoneme_error(predictions, gold_text)
    rtf = processing_time / audio_duration
    return {'accuracy': acc, 'fr': fr, 'rr': rr, 'eer': eer, 'pr': pr, 'llm_score': llm_score, 'utmos': utmos, 'per': per, 'rtf': rtf}

Common pitfalls

• Models often trade semantic sophistication for acoustic clarity, leading to lower UTMOS scores for complex or structured responses.
• Long-form outputs frequently suffer from PER degradation due to repeated phrases, omissions, or premature truncation.
• Multi-turn dialogue accuracy drops significantly compared to single-turn due to weak conversational history modeling.
• Code-switching inputs cause >33% performance drops due to poor cross-lingual semantic alignment.

Evidence (verbatim from paper)

For instances with explicit answers, we used objective accuracy, along with the following rate (FR) for instruction following, and the refusal rate (RR) for safety alignment. For open-ended questions, including creativity and single-round dialogue, we adopt the LLM evaluation score on a 1-5 scale. In the emotional empathy set, we defined the emotional empathy rate (EER), representing the proportion of the model’s response that demonstrates empathy toward the user in both semantics and acoustic tone. In the code-switching context and robustness set, we reported the preserve rate (PR), as the proportion of scores the model achieves under distracted or multilingual conditions relative to clean or monolingual conditions.

Citation

@misc{liu2025vocalbenchzh,
  title={VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2511.08230}
}

• arXiv: 2511.08230