Incompebench Eval

incompebench-eval

IncompeBench: A Permissively Licensed, Fine-Grained Benchmark for Music Information Retrieval — Clavié et al. (2026) (arXiv:2602.11941, 2026)

What this evaluates

This benchmark evaluates fine-grained music information retrieval by measuring how well models match audio tracks to diverse text queries. It probes the ability to capture nuanced musical attributes, handle negations, and rank candidates based on graded relevance rather than binary matches.

Datasets

• IncompeBench — total 128000; splits: test (128000); repo https://github.com/mixedbread-ai/incompebench-programs

Metrics

• graded relevance (0-3) (primary) — range: [0, 3]
  • A 4-point relevance scale where 0 = completely irrelevant, 1 = partially relevant, 2 = highly relevant, and 3 = fully relevant to every aspect of the query. Used to compute standard ranking metrics like nDCG and MRR.

Input / output format

Input: Text query + 30-second audio chunk (16kHz) for each candidate song.

Output: Graded relevance score (integer 0, 1, 2, or 3).

Scoring recipe

def compute_metric(predictions, gold):
    # predictions: list of graded relevance scores (0-3) for retrieved candidates
    # gold: list of graded relevance scores (0-3) for the same candidates
    # Compute Mean Reciprocal Rank (MRR) based on fully relevant (3) items
    for i, pred in enumerate(predictions):
        if pred == 3:
            return 1.0 / (i + 1)
    return 0.0

Common pitfalls

• Query negations (e.g., 'high BPM without guitar') require careful handling to avoid lenient matches.
• Candidate generation uses Reciprocal Rank Fusion across diverse models, which may bias ranking evaluation toward models used in the fusion step.
• LLM-generated annotations exhibit leniency; the benchmark releases 'Lenient' and 'Strict' variants to account for this noise.

Evidence (verbatim from paper)

The labelling process is conducted with Gemini 3 Pro... with fine-grained relevance levels, ranging from 0 (completely irrelevant) to 3 (fully relevant to every aspect of the query).

Citation

@misc{clavie2026incompebench,
  title={IncompeBench: A Permissively Licensed, Fine-Grained Benchmark for Music Information Retrieval},
  author={Clavié et al. (2026)},
  year={2026},
  note={arXiv:2602.11941}
}

• arXiv: 2602.11941