# Halluaudio Eval

> This benchmark probes the hallucination detection capabilities of Large Audio-Language Models (LALMs) across speech, environmental sound, and music domains. It systematically induces hallucinations using adversarial prompts and mixed-audio inputs to evaluate response correctness, affirmative bias, and refusal behavior beyond standard accuracy. Use when the user wants to benchmark on HalluAudio, or asks about evaluating this task. Reports Accuracy.

- Skill: `qhjqhj00/halluaudio-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/halluaudio-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/halluaudio-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/halluaudio-eval

---


# halluaudio-eval

> HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models — Zhao et al. (2026) (arXiv:2604.19300, 2026)

## What this evaluates

This benchmark probes the hallucination detection capabilities of Large Audio-Language Models (LALMs) across speech, environmental sound, and music domains. It systematically induces hallucinations using adversarial prompts and mixed-audio inputs to evaluate response correctness, affirmative bias, and refusal behavior beyond standard accuracy.

## Datasets

- **HalluAudio** — total ?; splits: test (-1); repo https://github.com/Feiyuzhao25/halluaudio

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Measures the fraction of prompts with well-defined ground truth answered correctly: $\text{Accuracy}_{d}=\frac{1}{|P_{d}|}\sum_{p\in P_{d}}\mathbf{1}{\hat{y}_{p}=y_{p}}$. Captures explicit hallucination cases where predictions contradict ground-truth audio evidence.
- `Yes-p Ratio` — range: [0, 1]
  - Diagnoses affirmative bias in binary responses: $\text{Yes-p Ratio}=\frac{\sum_{p\in P_{\text{binary}}}\mathbf{1}{\hat{y}_{p}=\text{Yes}}}{|P_{\text{binary}}|}$. Higher values indicate a tendency to produce unsupported positive responses.
- `Unrelated Ratio` — range: [0, 1]
  - Measures the proportion of incorrect binary predictions that are affirmative: $\text{Unrelated Ratio}=\frac{\sum_{p\in P_{\text{binary}}}\mathbf{1}{\hat{y}_{p}\neq y_{p}\wedge\hat{y}_{p}}}{\sum_{p\in P_{\text{binary}}}\mathbf{1}{\hat{y}_{p}\neq y_{p}}}$. Highlights cases where affirmative bias leads to semantically unrelated errors.
- `Conditional Accuracy` — range: [0, 1]
  - Evaluates accuracy specifically on binary prompts: $\text{Conditional Accuracy}=\frac{\sum_{p\in P_{\text{binary}}}\mathbf{1}{\hat{y}_{p}=y_{p}}}{|P_{\text{binary}}|}$. Reveals asymmetric decision behavior where models may perform well on positives but poorly on negatives.
- `False Refusal Rate (FRR)` — range: [0, 1]
  - Captures cases where models abstain despite valid prompts: $\text{FRR}_{d}=\frac{|{p\in P_{d}:\hat{y}_{p}\in\mathcal{R}}|}{|P_{d}|}$. Reflects over-conservative hallucination behavior and breakdowns in evidence-based decision making.

## Input / output format

**Input**: Audio input (speech, environmental sound, or music) paired with a text prompt. Prompts cover binary classification, multi-choice reasoning, attribute verification, and open-ended QA, often incorporating adversarial perturbations and mixed-audio contexts to induce hallucinations.

**Output**: Model's predicted response, typically a binary Yes/No, a classification label, or a free-text answer.

## Scoring recipe

```python
def compute_metrics(predictions, golds, domain='all'):
    P = [(p, g) for p, g in zip(predictions, golds) if g.domain == domain]
    acc = sum(1 for p, g in P if p == g) / len(P)
    binary_P = [(p, g) for p, g in P if g.task_type == 'binary']
    yes_ratio = sum(1 for p, _ in binary_P if p == 'Yes') / len(binary_P)
    wrong = [p for p, g in binary_P if p != g]
    unrelated_ratio = sum(1 for p in wrong if p == 'Yes') / len(wrong) if wrong else 0
    cond_acc = sum(1 for p, g in binary_P if p == g) / len(binary_P)
    refusal_set = {'refuse', 'cannot answer', 'i dont know'}
    frr = sum(1 for p, _ in P if p.lower() in refusal_set) / len(P)
    return {'accuracy': acc, 'yes_ratio': yes_ratio, 'unrelated_ratio': unrelated_ratio, 'conditional_accuracy': cond_acc, 'frr': frr}
```

## Common pitfalls

- Treating a high Yes-pred ratio as good performance, whereas it often indicates unsupported affirmative bias rather than evidence-grounded reasoning.
- Ignoring False Refusal Rate (FRR) as a distinct hallucination mode; models may abstain despite clear acoustic evidence, reflecting over-conservative decision policies rather than safety alignment.
- Aggregating accuracy across speech, sound, and music domains without accounting for task-specific structural vs. semantic hallucination patterns, which vary significantly across domains.

## Evidence (verbatim from paper)

> Accuracy measures the fraction of prompts with well-defined ground truth that are answered correctly by LALMs: $\text{Accuracy}_{d}=\frac{1}{|P_{d}|}\sum_{p\in P_{d}}\mathbf{1}{\hat{y}_{p}=y_{p}}$

## Citation

```bibtex
@misc{zhao2026halluaudio,
  title={HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2604.19300}
}
```

- arXiv: 2604.19300