Image To Music Eval

image-to-music-eval

Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approach — Zijian Zhao et al. (2025) (arXiv:2509.22378, 2025)

What this evaluates

Evaluates the capability of generative models to produce symbolic music (ABC notation) that aligns with a given input image. It probes both the intrinsic musical quality of the generated output and the semantic/emotional consistency between the source image and the resulting composition.

Datasets

• Image-to-Music test set [[30]] — total ?; splits: test (-1); repo https://github.com/RS2002/Image2Music

Metrics

• Music Quality Level (primary) — range: [1, 7]
  • Arithmetic mean of scores across Overall, Melody, Rhythm, Authenticity, and Harmony dimensions.
• Music-Image Consistency — range: [1, 7]
  • Arithmetic mean of scores across Overall, Semantics, and Emotion dimensions.
• SongEval Average — range: varies
  • Automated score from the SongEval benchmark assessing Coherence, Musicality, Memorability, Clarity, and Naturalness.

Input / output format

Input: Single input image (RGB).

Output: Symbolic music representation in ABC notation.

Scoring recipe

def compute_scores(predictions, gold_images, evaluator='human'):
    scores = []
    for img, music in zip(gold_images, predictions):
        if evaluator == 'human':
            # 31 participants score each pair on 1-7 scale
            score = sum([p.score(img, music) for p in participants]) / len(participants)
        elif evaluator == 'vlm':
            score = grok4_judge.score(img, music)
        elif evaluator == 'songeval':
            score = songeval_model.score(music)
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Audio-based methods cannot be directly evaluated with LLM judges; they must be converted to ABC notation first, which may introduce transcription errors.
• Human evaluation is limited to 3 pairs per method to prevent participant fatigue, making it difficult to include extensive ablation studies in the human loop.
• Traditional deterministic metrics like PCE and EBR are ignored because they require ground truth and their optimality direction depends on music style.

Evidence (verbatim from paper)

For human evaluation, we ask 31 participants to assess three image-music pairs for each method (informed consent was obtained), scoring them from 1 to 7 based on the following metrics inspired by [[31], [32]]: (i) music quality level, including overall quality, melody, rhythm, authenticity, and harmony; (ii) music-image consistency, encompassing overall correspondence, semantic consistency, and emotional consistency.

Citation

@misc{zhao2025zeroeffort,
  title={Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approach},
  author={Zijian Zhao et al. (2025)},
  year={2025},
  note={arXiv:2509.22378}
}

• arXiv: 2509.22378