Mecat Eval

mecat-eval

MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks — Niu et al. (2025) (arXiv:2507.23511, 2025)

What this evaluates

This benchmark evaluates fine-grained audio understanding by testing models on generating detailed, multi-perspective captions and answering probing questions across diverse acoustic domains. It specifically probes a model's ability to distinguish between speech, music, and sound events, reason about acoustic scenes, and assess technical audio quality without relying on generic descriptions.

Datasets

• MECAT — total 20000; splits: test (20000); repo https://github.com/xiaomi-research/mecat

Metrics

• DATE (primary) — range: [0, 1]
  • DATE combines weighted semantic similarity (with penalties for generic terms) and cross-sample discriminability. The overall evaluation score is computed as a weighted average of caption and QA scores: Score_Cap = 0.4S_Systemic + 0.4S_Content-Specific + 0.2S_Content-Unrelated, where S_Systemic = 0.8S_Long + 0.2S_Short and S_Content-Specific = 0.6S_Speech + 0.3S_Music + 0.1S_Sound. Score_QA is the unweighted mean of six cognitive sub-categories (DP, SC, QAS, ER, IJ, AC).

Input / output format

Input: Audio clip (maximum 10 seconds duration) paired with a specific question (for QA task) or provided as-is (for captioning task).

Output: Text string containing a caption (short, long, domain-specific, or content-unrelated) or a direct answer to the posed question.

Scoring recipe

S_long = DATE(pred, refs_long)
S_short = DATE(pred, refs_short)
S_speech = DATE(pred, refs_speech)
S_music = DATE(pred, refs_music)
S_sound = DATE(pred, refs_sound)
S_unrelated = DATE(pred, refs_unrelated)
S_DP = DATE(pred, refs_DP)
S_SC = DATE(pred, refs_SC)
S_QAS = DATE(pred, refs_QAS)
S_ER = DATE(pred, refs_ER)
S_IJ = DATE(pred, refs_IJ)
S_AC = DATE(pred, refs_AC)
S_systemic = 0.8 * S_long + 0.2 * S_short
S_content_specific = 0.6 * S_speech + 0.3 * S_music + 0.1 * S_sound
Score_Cap = 0.4 * S_systemic + 0.4 * S_content_specific + 0.2 * S_unrelated
Score_QA = (S_DP + S_SC + S_QAS + S_ER + S_IJ + S_AC) / 6
return Score_Cap, Score_QA

Common pitfalls

• Models often generate generic captions that score poorly on DATE due to the explicit penalty for non-specific terms.
• Evaluators must ensure domain-specific captions explicitly state when a domain (e.g., silence or music) is absent, as the benchmark requires negative assertions.
• The weighted aggregation for captioning heavily prioritizes systemic and content-specific captions (0.8 weight each), so neglecting short captions or content-unrelated acoustic quality will disproportionately lower the final score.

Evidence (verbatim from paper)

It addresses the gap in current benchmarks by introducing DATE, a novel evaluation metric that combines weighted semantic similarity (penalizing generic terms) with cross-sample discriminability to robustly distinguish between superficial and contextually rich model outputs, enabling more accurate assessment of perceptual fidelity in audio understanding.

Citation

@misc{niu2025mecat,
  title={MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks},
  author={Niu et al. (2025)},
  year={2025},
  note={arXiv:2507.23511}
}

• arXiv: 2507.23511