Deaf Acoustic Faithfulness Eval

deaf-acoustic-faithfulness-eval

DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models — Xiong et al. (2026) (arXiv:2603.18048, 2026)

What this evaluates

This benchmark probes the acoustic faithfulness of Audio Multimodal Large Language Models (Audio MLLMs) by measuring how reliably they attend to acoustic cues (emotional prosody, background sounds, speaker identity) when faced with conflicting textual semantics or misleading prompts. It specifically diagnoses the tendency of models to prioritize text over audio (text dominance) under progressive levels of interference.

Datasets

• DEAF — total 2700; splits: test (2700)

Metrics

• Acoustic Robustness Score (ARS) (primary) — range: percent
  • Percentage score (0–100%) representing the proportion of instances where the model correctly identifies the acoustic property despite conflicting textual cues or misleading prompts. Higher values indicate stronger acoustic grounding.
• Environment Discrimination Index (EDI) — range: other
  • Difference in accuracy between coarse cross-category environmental discrimination and fine-grained within-category discrimination. Positive values indicate better coarse discrimination; negative values indicate within-category accuracy exceeds cross-category.

Input / output format

Input: A 16 kHz WAV audio clip paired with a text question/prompt. Prompts vary by conflict level (L1: semantic conflict only, L2: misleading prompt only, L3: dual interference) and mention type (explicit vs. implicit).

Output: Open-ended textual response.

Scoring recipe

For each sample i:
  response_i = model.generate(audio_i, prompt_i)
  correct_i = llm_judge.verify(response_i, gold_acoustic_property)
# ARS is the percentage of correct responses for a given conflict type/level
ARS = (sum(correct_i) / N) * 100
# EDI measures environmental discrimination granularity
EDI = accuracy(coarse_cross_category) - accuracy(fine_within_category)

Common pitfalls

• Models frequently exhibit 'text dominance,' answering based on the prompt's semantic content rather than the audio, which artificially deflates ARS scores.
• Performance is highly sensitive to the conflict level (L1 vs L2 vs L3) and mention type (explicit vs implicit), so aggregating across levels without stratification masks critical failure modes.
• The LLM-as-judge evaluation introduces potential bias or variance, necessitating multiple runs (3 in this case) and averaging to stabilize results.

Evidence (verbatim from paper)

Each sample is evaluated in a zero-shot setting, where the model receives a 16 kHz WAV audio clip and a question, and generates an open-ended textual response. All evaluations are conducted in independent sessions and repeated three times, with the average results reported. Table 3: Acoustic Robustness Score (ARS, %) across conflict types and levels. Higher values indicate stronger acoustic grounding under semantic conflict.

Citation

@misc{xiong2026deaf,
  title={DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models},
  author={Xiong et al. (2026)},
  year={2026},
  note={arXiv:2603.18048}
}

• arXiv: 2603.18048