Bass Eval

bass-eval

BASS: Benchmarking Audio LMs for Musical Structure and Semantic Reasoning — Jang et al. (2026) (arXiv:2602.04085, 2026)

What this evaluates

This benchmark evaluates audio language models on musical understanding and semantic reasoning across four domains: structural segmentation, lyric transcription, musicological analysis, and artist collaboration. It probes the model's ability to process long-form audio and perform temporal, hierarchical, and musicological reasoning over vocal and structural attributes.

Datasets

• BASS — total 2658; splits: test (-1); repo https://github.com/minjang10/bass_music_benchmark

Metrics

• IWER (Normalized Word Error Rate) (primary) — range: [0, 1]
  • IWER = 1 / (1 + WER), where WER is the word error rate averaged across all sections. The score is bounded between 0.0 and 1.0 and reported as a percentage.
• IoU (Intersection over Union) — range: [0, 1]
  • Intersection counts only if predicted and reference section labels match. Divided by the union of predicted and reference sections. Reported as a percentage.
• EMA_N (Normalized Exact Match Accuracy) — range: [-∞, 1]
  • EMA_N = (EMA - R) / (1 - R), where EMA is exact match accuracy and R is random guessing accuracy. Predictions within a 3-second offset are correct for timestamp/duration questions. Reported as a percentage.

Input / output format

Input: A natural language instruction prompt and an audio file.

Output: Free-form text generation, except for Musicological Analysis tasks which use a multiple-choice format.

Scoring recipe

# For Lyric Transcription:
wer = compute_wer(reference_lyrics, predicted_lyrics)
iwer = 1 / (1 + wer)
score_pct = iwer * 100

# For Structural Segmentation:
iou = compute_iou(reference_sections, predicted_sections, label_match=True)
score_pct = iou * 100

# For other tasks:
ema = compute_exact_match_accuracy(predictions, references, timestamp_offset=3.0)
ema_n = (ema - random_guess_accuracy) / (1 - random_guess_accuracy)
score_pct = ema_n * 100

Common pitfalls

• WER can exceed 1.0 due to excessive insertions and background noise in long-form musical audio, making it unbounded and requiring inversion for aggregation.
• Naively averaging EMA across tasks is invalid because different tasks have different random chance accuracies; normalization against random guessing (R) is strictly required.
• Multiple-choice tasks require majority voting over four independent runs to determine final performance, not a single pass.

Evidence (verbatim from paper)

For Structural Lyric Transcription, we use word error rate (WER). For questions involving multiple sections, reference and predicted sections are optimally matched, and the final WER is the average across all sections. For Structural Segmentation, performance is measured using intersection over union (IoU), where an overlap counts as an intersection only if the predicted section labels match the reference. For all other tasks, we use exact match accuracy (EMA). For questions requiring timestamps or durations, predictions within a 3-second offset are considered correct. ... Hence, we invert WER so that higher scores corresponds to better performance, IWER=1/(1+WER), bounding the score between 0.0 and 1.0. To address the second issue, we normalize EMA to represent models’ improvement over random guessing. Specifically, we report EMA_N=(EMA-R)/(1-R) where R is random guessing accuracy.

Citation

@misc{jang2026bass,
  title={BASS: Benchmarking Audio LMs for Musical Structure and Semantic Reasoning},
  author={Jang et al. (2026)},
  year={2026},
  note={arXiv:2602.04085}
}

• arXiv: 2602.04085