Hello Chat Eval

hello-chat-eval

Hello-Chat: Towards Realistic Social Audio Interactions — Hou et al. (2026) (arXiv:2602.23387, 2026)

What this evaluates

Evaluates an end-to-end Large Audio Language Model's capabilities in audio understanding (ASR, QA, translation, reasoning, emotion/event recognition, instruction following) and text-to-speech synthesis (naturalness, intelligibility, speaker similarity).

Datasets

• AIShell, WeNet, LibriSpeech — total ?; splits: test (-1)
• AlpacaEval, LLaMA Questions, Web Questions — total ?; splits: test (-1)
• Synthetic multilingual (Claude-generated) — total ?; splits: test (-1)
• MMAU-Mini — total ?; splits: test (-1)
• EmoBox — total ?; splits: test (-1)
• AudioSet, CochlScene — total ?; splits: test (-1)
• Seed-TTS-Eval (Chinese) — total ?; splits: test (-1)

Metrics

• WER/CER (primary) — range: percent
  • Word/Character Error Rate: edit distance between predicted and reference text divided by reference length. Lower values indicate better recognition accuracy.
• GPT-5 score — range: [0, 10]
  • LLM-as-a-judge scoring of answer correctness and semantic relevance on a 1-10 scale.
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted labels or answers (used for MMAU, SER, AED, Instruction Following). Higher values indicate better performance.
• CMOS (primary) — range: [1, 5]
  • Conversational-style Mean Opinion Score: human evaluators rate naturalness and prosodic appropriateness on a 1-5 Likert scale.
• SS — range: [-1, 1]
  • Speaker Similarity: cosine similarity between WavLM embeddings of synthesized and reference audio.

Input / output format

Input: Audio clips (speech, multilingual speech, audio events) or text converted to speech via TTS; for TTS evaluation, reference text and speaker embedding/reference audio.

Output: Text responses (transcriptions, answers, translations, labels) or synthesized audio waveforms.

Scoring recipe

# ASR
wer = edit_distance(pred_text, ref_text) / len(ref_text)
# QA/Translation
score = gpt5_model.evaluate(pred_text, ref_text, prompt="correctness and relevance")
# MMAU/SER/AED/Instr
acc = 1 if pred_label == gold_label else 0
# TTS
cer = edit_distance(paraformer_transcribe(audio), ref_text) / len(ref_text)
ss = cosine_similarity(wavlm_embed(audio), wavlm_embed(ref_audio))
cmos = mean([human_evaluator.score(audio, scale=1..5) for _ in range(15)])

Common pitfalls

• GPT-5 scoring for QA and translation is non-deterministic and heavily depends on the specific LLM version and prompt template used.
• CMOS is a subjective human evaluation requiring 15 native speakers, making exact reproduction difficult across different labs.
• ASR evaluation mixes Chinese and English corpora; reporting a single WER/CER without language breakdown can mask performance disparities.

Evidence (verbatim from paper)

Speech intelligibility is assessed using CER, computed by transcribing synthesized speech with the Paraformer model and comparing it to the reference text. Speaker similarity (SS) is measured as the cosine similarity between speaker embeddings extracted by a WavLM-based speaker verification model, following the official Seed-TTS-Eval protocol. To evaluate the model’s performance in real-world interaction, we conducted a subjective assessment using the Conversational-style Mean Opinion Score (CMOS).

Citation

@misc{hou2026hellochat,
  title={Hello-Chat: Towards Realistic Social Audio Interactions},
  author={Hou et al. (2026)},
  year={2026},
  note={arXiv:2602.23387}
}

• arXiv: 2602.23387