Kimi Audio Eval

kimi-audio-eval

Kimi-Audio Technical Report — KimiTeam et al. (2025) (arXiv:2504.18425, 2025)

What this evaluates

Evaluates an audio foundation model's capabilities across automatic speech recognition, general audio understanding, audio-to-text conversational reasoning, and end-to-end speech conversation.

Datasets

• LibriSpeech — total ?; splits: test-clean (-1), test-other (-1)
• FLEURS — total ?; splits: en (-1)
• AISHELL-1 — total ?; splits: train (-1)
• AISHELL-2 — total ?; splits: train (-1)
• WenetSpeech — total ?; splits: test-meeting (-1), test-net (-1)
• Kimi-ASR Internal Testset — total ?; splits: subset1 (-1), subset2 (-1)
• MMAU — total ?; splits: sound (-1), speech (-1)
• ClothoAQA — total ?; splits: dev (-1)
• VocalSound — total ?; splits: test (-1)
• Nonspeech7k — total ?; splits: test (-1)
• MELD — total ?; splits: test (-1)
• TUT2017 — total ?; splits: test (-1)
• CochlScene — total ?; splits: dev (-1)
• OpenAudioBench — total ?; splits: test (-1)
• VoiceBench — total ?; splits: test (-1)

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • 100 * (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, N=total words in reference. Calculated using a standardized implementation based on Qwen-2-Audio with consistent text normalization.
• Accuracy / Benchmark-specific scores — range: percent
  • Percentage of correct predictions for classification tasks; LLM-judged or exact-match scores for open-ended QA and reasoning tasks.
• Human Rating — range: [1, 5]
  • Average score on a 1-5 Likert scale across dimensions: speed control, accent control, emotion control, empathy, and style control.

Input / output format

Input: Raw audio files (speech or environmental) paired with task-specific text prompts or questions.

Output: Text transcripts (ASR), categorical labels or free-form text answers (understanding/chat), or synthesized audio waveforms (conversation).

Scoring recipe

def score_asr(preds, golds):
    # Standardized WER with consistent text normalization
    return sum(wer(p, g) for p, g in zip(preds, golds)) / len(golds)

def score_understanding(preds, golds, task):
    if task in ['MMAU', 'VocalSound', 'Nonspeech7k', 'MELD', 'TUT2017', 'CochlScene']:
        return accuracy(preds, golds)
    else:
        return llm_judge_score(preds, golds, judge='GPT-4o-mini')

def score_conversation(preds, golds):
    # Human evaluation on 1-5 scale
    return average_human_rating(preds)

Common pitfalls

• Inconsistent WER calculation due to varying text normalization practices across different toolkits.
• Relying solely on exact string matching for open-ended audio QA fails to capture semantic correctness of complex LLM responses.
• High sensitivity of model performance to inference parameters like decoding temperature, system prompts, and task prompts.

Evidence (verbatim from paper)

We implement a standardized WER calculation (based on Qwen-2-Audio) and integrate GPT-4o-mini as an intelligent judge (following chen2024voicebench) for tasks like audio question answering. This approach overcomes the limitations of inconsistent metrics and simplistic string matching, enabling fair comparison.

Citation

@misc{kimiteam2025kimi-audio,
  title={Kimi-Audio Technical Report},
  author={KimiTeam et al. (2025)},
  year={2025},
  note={arXiv:2504.18425}
}

• arXiv: 2504.18425