Must RAG Eval

must-rag-eval

MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation — Kwon et al. (2025) (arXiv:2507.23334, 2025)

What this evaluates

This evaluation probes a model's ability to answer music-specific factual and contextual questions using retrieval-augmented generation. It measures accuracy on both in-domain artist metadata and out-of-domain music knowledge across multiple-choice formats.

Datasets

• ArtistMus — total ?; splits: Seen (-1), Unseen (-1)
• TrustMus — total 400; splits: Ppl (100), IT (100), GFT (100), CH (100)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. A response is marked correct if it matches the ground truth answer and adheres to the expected output format.

Input / output format

Input: A multiple-choice question, optionally accompanied by retrieved context passages. For zero-shot baselines, only the question is provided.

Output: The model must generate the correct answer choice, strictly adhering to the expected format without extra conversational text.

Scoring recipe

correct = 0
for q, gold, pred in dataset:
    if pred.strip() == gold.strip() and format_valid(pred):
        correct += 1
return (correct / len(dataset)) * 100

Common pitfalls

• Responses that contain the correct answer but include extra conversational filler or deviate from the strict expected format are automatically scored as incorrect.
• The 'Seen' vs 'Unseen' split is defined by artist presence in the training set, not question difficulty, which can skew baseline comparisons if not accounted for.

Evidence (verbatim from paper)

All evaluations use a multiple-choice QA format. Following [[11]], we consider a response incorrect if it deviates from the expected format.

Citation

@misc{kwon2025must_rag,
  title={MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation},
  author={Kwon et al. (2025)},
  year={2025},
  note={arXiv:2507.23334}
}

• arXiv: 2507.23334