Balsa Audio Eval

balsa-audio-eval

From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data — Kuan et al. (2025) (arXiv:2505.20166, 2025)

What this evaluates

Evaluates audio-language alignment, reasoning, and instruction-following capabilities of audio-aware large language models. It probes the model's ability to answer audio-based questions, perform semantic reasoning, detect hallucinations, and follow complex multimodal instructions.

Datasets

• ClothoAQA — total ?; splits: test (-1)
• Synonym-Hypernym Test — total ?; splits: test (-1)
• MMAU — total ?; splits: test (-1)
• MMAR — total ?; splits: test (-1)
• SAKURA — total ?; splits: test (-1)
• Audio Hallucination Benchmark — total ?; splits: test (-1)
• Instruction-Following Benchmark — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted answers out of total instances. Computed as exact match between extracted model output and ground truth option.
• weighted F1 score (primary) — range: [0, 1]
  • F1 score averaged across classes, weighted by class support (number of true instances per class). Used for multi-class classification tasks.
• micro-averaged accuracy — range: [0, 1]
  • Total correct predictions divided by total predictions, equivalent to overall accuracy across all samples and classes.
• weighted precision — range: [0, 1]
  • Weighted average of per-class precision, where weights correspond to class support.
• weighted recall — range: [0, 1]
  • Weighted average of per-class recall, where weights correspond to class support.
• proportion of 'Yes' responses — range: [0, 1]
  • Fraction of model outputs that exactly match the string 'Yes', used as an auxiliary reference for hallucination detection.

Input / output format

Input: Audio clips paired with text instructions or multiple-choice questions.

Output: Free-form text responses generated via greedy decoding with a maximum length of 512 tokens.

Scoring recipe

def score(predictions, gold_options, metric_type):
    extracted = [extract_regex(p) for p in predictions]
    if metric_type == 'accuracy':
        return sum(1 for e, g in zip(extracted, gold_options) if e == g) / len(gold_options)
    elif metric_type == 'weighted_f1':
        return f1_score(gold_options, extracted, average='weighted')
    elif metric_type == 'yes_proportion':
        return sum(1 for e in extracted if e == 'Yes') / len(extracted)
    return 0.0

Common pitfalls

• Regex-based answer extraction may fail on unstructured or verbose model outputs, automatically marking them as incorrect.
• Answer options are randomized before evaluation to mitigate positional bias, so models must rely on content rather than option order.
• Greedy decoding with a fixed 512-token limit is used for all baselines, which may disadvantage models that benefit from sampling or longer generation.

Evidence (verbatim from paper)

We report the weighted F1 score for multi-class classification tasks. For other tasks with their own evaluation protocols, we follow established methodologies. Accuracy is adopted as the primary metric for ClothoAQA, which comprises both binary and non-binary classification tasks. Following previous studies, we compute overall accuracy and F1 scores for questions where the correct answer is “Yes” or “No”. As an additional reference, we also report the proportion of cases where the model responds with “Yes”.

Citation

@misc{kuan2025balsa,
  title={From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data},
  author={Kuan et al. (2025)},
  year={2025},
  note={arXiv:2505.20166}
}

• arXiv: 2505.20166