qhjqhj00 brace-main-eval
BRACE: A Benchmark for Robust Audio Caption Quality Evaluation — Guo et al. (2025) (arXiv:2512.10403, 2025)
What this evaluates
Evaluates the ability of audio-language models to align audio with captions and distinguish caption quality across different generation sources (human-human, human-machine, machine-machine). It probes fine-grained semantic and syntactic alignment capabilities under realistic captioning conditions.
Datasets
- BRACE-Main — total ?; splits: test (-1); repo https://github.com/HychTus/BRACE_Evaluation
Metrics
F1-score(primary) — range: [0, 1]- Harmonic mean of precision and recall for caption quality alignment or preference prediction between caption pairs.
Input / output format
Input: Audio clip paired with two captions (Human-Human, Human-Machine, or Machine-Machine pairs) for quality comparison or alignment scoring.
Output: Model outputs a similarity score, preference choice, or quality rating for the caption pair.
Scoring recipe
def compute_f1(predictions, gold):
tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
Common pitfalls
- CLAP models struggle with fine-grained Human-Human and Machine-Machine caption comparisons, often failing to identify subtle quality differences.
- Position bias in LALMs causes them to favor caption_0 or caption_1 regardless of actual content, especially under complex prompts.
Evidence (verbatim from paper)
On BRACE-Main, the best-performing model LAION-CLAP achieves an F1-score of 70.01, while others range from ~55 to 70 depending on architecture and subset.
Citation
@misc{guo2025brace,
title={BRACE: A Benchmark for Robust Audio Caption Quality Evaluation},
author={Guo et al. (2025)},
year={2025},
note={arXiv:2512.10403}
}
- arXiv: 2512.10403