Vcb Bench Eval

vcb-bench-eval

VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents — Hu et al. (2025) (arXiv:2510.11098, 2025)

What this evaluates

VCB Bench evaluates audio-grounded large language models on instruction following with speech-level controls, knowledge reasoning, and robustness under real-world acoustic perturbations. It probes how well models understand and generate spoken responses in Chinese and English using authentic human speech rather than synthetic data.

Datasets

• VCB Bench — total ?; splits: test (-1); repo https://github.com/193746/VCB-Bench-Evalkit

Metrics

• 1-5 scale score (primary) — range: [1, 5]
  • LLM judges (Gemini-2.5-Pro or GPT-4o) assign a numerical score from 1 to 5 based on response quality. For reference-based QA, it is a binary 'Yes'/'No' judgment.
• MOS — range: [1, 5]
  • Mean Opinion Score rated by eight human experts on a 1-5 scale for speech instruction following tasks.
• StoryCloze accuracy — range: [0, 1]
  • Computed by comparing negative log-likelihoods of correct vs. incorrect story endings; the ending with lower NLL is selected as correct.

Input / output format

Input: Audio prompts containing spoken instructions or context (sometimes accompanied by text). For SC, audio/text story contexts with two possible endings.

Output: Spoken audio responses. For SC, a binary selection of the correct ending based on log-likelihood.

Scoring recipe

def score_vcb_bench(predictions, gold, task_type):
    if task_type == 'SC':
        nll_correct = compute_nll(predictions['correct_ending'], gold['context'])
        nll_incorrect = compute_nll(predictions['incorrect_ending'], gold['context'])
        return 1.0 if nll_correct < nll_incorrect else 0.0
    elif task_type == 'MTD':
        turn_scores = [judge_score(turn_pred, turn_gold) for turn_pred, turn_gold in zip(predictions, gold)]
        final_weight = 0.5
        earlier_weight = 0.5 / (len(turn_scores) - 1)
        return turn_scores[-1] * final_weight + sum(turn_scores[:-1]) * earlier_weight
    else:
        judge = Gemini25Pro if task_type in ['SIF', 'SIF-En'] else GPT4o
        if task_type == 'reference-based':
            return 1.0 if judge(predictions, gold) == 'Yes' else 0.0
        return judge(predictions, gold)

Common pitfalls

• Using synthetic or TTS-generated speech instead of authentic human recordings, which inflates robustness scores and misrepresents real-world performance.
• Ignoring the 50% weight on the final dialogue turn in MTD evaluation, leading to inaccurate long-context capability assessment.
• Relying solely on text-only evaluation for audio generation tasks without ASR transcription, which fails to capture audio clarity and text-speech alignment issues.

Evidence (verbatim from paper)

In open-ended QA, both Gemini and GPT assign a numerical score on a 1–5 scale, whereas for reference-based QA, they output a binary judgment of "Yes" or "No." For the SC task, we assess a subset of pre-trained base models: Baichuan-Audio-Base, Kimi-Audio-Base, Qwen2-Audio-Base, and Step-Audio 2 mini-Base. Following the StoryCloze evaluation protocol, we compute the negative log-likelihood for both the correct and incorrect endings, with model selection determined by comparing these two values. For SIF tasks, six performing models undergo further Mean Opinion Score (MOS) evaluation.

Citation

@misc{hu2025vcbbench,
  title={VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents},
  author={Hu et al. (2025)},
  year={2025},
  note={arXiv:2510.11098}
}

• arXiv: 2510.11098