Muchin Eval

muchin-eval

MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music — Wang et al. (2024) (arXiv:2402.09871, 2024)

What this evaluates

Evaluates language models' ability to generate structured Chinese lyrics from music descriptions and to understand music audio by generating descriptive tags. It probes alignment with public (amateur) vs. professional musical perception and semantic similarity in Chinese.

Datasets

• MuChin — total ?; splits: test (-1)

Metrics

• Overall Score (primary) — range: [0, 100]
  • Weighted average of six Gestalt-based string similarity scores: Song (0.10), Section (0.325), Phrase (0.175), Word (0.20), Rhyming Fitting Accuracy (0.20), and Rhyming Proportion Reasonableness (0.10).
• Semantic Similarity Score — range: [0, 1]
  • Average cosine similarity between BGE-large-zh-v1.5 embeddings of generated descriptive tags and ground truth tags across 10 predefined dimensions.

Input / output format

Input: For lyric generation: Chinese text describing music. For music understanding: Audio files of music tracks.

Output: For lyric generation: Structured lyrics with section labels and rhyming patterns. For music understanding: 10 descriptive tags corresponding to predefined musical dimensions.

Scoring recipe

def calc_overall_score(gen_lyrics, gt_lyrics):
    song_sim = gestalt_similarity(gen_lyrics, gt_lyrics, level='song')
    section_sim = gestalt_similarity(gen_lyrics, gt_lyrics, level='section')
    phrase_sim = gestalt_similarity(gen_lyrics, gt_lyrics, level='phrase')
    word_sim = gestalt_similarity(gen_lyrics, gt_lyrics, level='word')
    rhyme_fit = rhyme_fitting_accuracy(gen_lyrics, gt_lyrics)
    rhyme_prop = rhyme_proportion_reasonableness(gen_lyrics, gt_lyrics)
    return (0.10*song_sim + 0.325*section_sim + 0.175*phrase_sim + 
            0.20*word_sim + 0.20*rhyme_fit + 0.10*rhyme_prop)

def calc_semantic_score(gen_tags, gt_tags):
    emb_gen = bge_large_zh_v1_5.encode(gen_tags)
    emb_gt = bge_large_zh_v1_5.encode(gt_tags)
    sims = [cosine_sim(g, t) for g, t in zip(emb_gen, emb_gt)]
    return mean(sims)

Common pitfalls

• Uses Gestalt algorithm for string matching instead of standard NLP metrics like BLEU or ROUGE.
• Semantic similarity relies specifically on BGE-large-zh-v1.5 embeddings, not generic sentence transformers.
• Evaluation prioritizes structural and rhyming accuracy over subjective lyrical creativity.

Evidence (verbatim from paper)

Semantic Similarity Score. The BGE model, as a general word vector embedding model, has demonstrated impressive performance on various tasks. We utilize the bge-large-zh-v1.5 model to calculate the semantic similarity between the generated and original tags.

Citation

@misc{wang2024muchin,
  title={MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2402.09871}
}

• arXiv: 2402.09871