Fun Audio Chat Eval

fun-audio-chat-eval

Fun-Audio-Chat Technical Report — Tongyi Fun Team et al. (2025) (arXiv:2512.20156, 2025)

What this evaluates

Evaluates a large audio language model's capabilities across spoken question answering, audio understanding, speech recognition, function calling, and instruction following. It probes the model's ability to process speech inputs, generate text/speech outputs, and adhere to complex voice instructions while maintaining speech quality and safety.

Datasets

• VoiceBench — total ?; splits: test (-1)
• OpenAudioBench — total ?; splits: test (-1); HF baichuan-inc/OpenAudioBench
• UltraEval-Audio — total ?; splits: test (-1); repo https://github.com/OpenBMB/UltraEval-Audio
• MMAU — total ?; splits: test (-1)
• MMAU-Pro — total ?; splits: test (-1)
• MMSU — total ?; splits: test (-1)
• Librispeech — total ?; splits: test (-1)
• Common Voice — total ?; splits: test (-1)
• Speech-ACEBench — total ?; splits: test (-1)
• Speech-BFCL — total ?; splits: test (-1)
• Speech-SmartInteract — total ?; splits: test (-1); HF FunAudioLLM/SpeechFCEval
• VStyle — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Calculated as the percentage of correctly predicted answers out of the total number of instances. Used for close-ended QA, audio understanding, and function calling tasks.
• G-Eval — range: percent
  • An LLM-based evaluation metric that scores the quality of open-ended responses against reference answers or criteria. Scores are normalized to percentages in this paper.
• ASR-WER — range: percent
  • Word Error Rate computed by transcribing the model-generated speech using Whisper-v3-large and comparing it against the ground-truth text. Formula: (S+D+I)/N.
• UTMOS — range: [1, 5]
  • Self-reported Mean Opinion Score for overall speech quality, rated on a 1-5 scale.
• Refusal Rate — range: percent
  • Percentage of safety-benchmark prompts where the model correctly refuses to generate a harmful or unsafe response.
• LALM evaluation score — range: [1, 5]
  • Scores assigned by a Large Audio Language Model acting as a judge, evaluating responses on a 1-5 scale across dimensions like acoustic attributes, instruction following, role-play, and empathy.

Input / output format

Input: Speech audio input paired with text instructions or questions.

Output: Text responses and/or synthesized speech audio.

Scoring recipe

def compute_metrics(predictions, gold, task_type):
    if task_type == 'close_ended_qa':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'open_ended_qa':
        return g_eval_score(predictions, gold)  # LLM-based
    elif task_type == 'safety':
        return refusal_rate(predictions)
    elif task_type == 'speech_quality':
        asr_transcripts = whisper_v3_large.predict(predictions)
        wer = word_error_rate(asr_transcripts, gold_text)
        utmos = utmos_score(predictions)
        return wer, utmos
    elif task_type == 'function_calling':
        return execute_and_check(predictions, gold)
    elif task_type == 'instruction_following':
        return lalm_judge_score(predictions, gold)
    return None

Common pitfalls

• G-Eval relies on an external LLM judge, making results sensitive to the judge model/version and potentially non-deterministic across runs.
• ASR-WER is computed using Whisper-v3-large specifically; results are not directly comparable to WER values measured with other ASR models or versions.
• The internal test set for voice empathy is proprietary and not publicly available, limiting full reproducibility of those specific metrics.
• UTMOS is reported as self-evaluated scores rather than independently verified by third-party human listeners.

Evidence (verbatim from paper)

Evaluations adhere to the established protocols for each respective benchmark. For S→T and S→S evaluations on Spoken Question Answering benchmarks, we use different metrics depending on the task type: (1) Accuracy is used for close-ended QA tasks including Llama Q., Reasoning QA, TriviaQA, Web Q., SD-QA, MMSU, OpenBookQA, and IFEval; (2) G-Eval is used for open-ended QA tasks including AlpacaEval and CommonEval, which employs LLM-based evaluation to assess response quality; (3) Refusal Rate is reported for AdvBench to measure safety compliance.

Citation

@misc{tongyifunteam2025funaudiachat,
  title={Fun-Audio-Chat Technical Report},
  author={Tongyi Fun Team et al. (2025)},
  year={2025},
  note={arXiv:2512.20156}
}

• arXiv: 2512.20156