Muspike Eval

muspike-eval

MuSpike: A Benchmark and Evaluation Framework for Symbolic Music Generation with Spiking Neural Networks — Liang et al. (2025) (arXiv:2508.19251, 2025)

What this evaluates

Evaluates the quality of symbolic music generation by spiking neural networks across multiple datasets. It assesses both objective statistical properties (pitch, rhythm, harmony) and subjective cognitive/perceptual dimensions (fluency, emotion, impression, autobiographical association).

Datasets

• JSB Chorales — total ?; splits: unspecified (-1)
• POP909 — total ?; splits: unspecified (-1)
• Lakh MIDI — total ?; splits: unspecified (-1)
• EMOPIA — total ?; splits: unspecified (-1)
• XMIDI — total ?; splits: unspecified (-1)

Metrics

• Pitch Count (PC) — range: other
  • Measures the number of distinct pitches in a generated sequence. Detailed calculation provided in Table A3 of the Appendix.
• Average inter-onset interval (IOI) — range: other
  • Calculates the average time between consecutive note onsets to assess rhythmic spacing. Detailed calculation provided in Table A3 of the Appendix.
• Pitch consonance score (PCS) — range: [0, 1]
  • Evaluates harmonic harmony by measuring the consonance of pitch combinations. Detailed calculation provided in Table A3 of the Appendix.
• Impression — range: percent
  • Cognitive-level metric capturing the overall aesthetic impact of the generated music on the listener (Q11).
• Autobiographical association — range: percent
  • Cognitive-level metric measuring the extent to which the music evokes personal memories or experiences (Q12).
• Personal preference (primary) — range: percent
  • Cognitive-level metric indicating the listener's overall liking or preference for the generated piece (Q13).

Input / output format

Input: MIDI sequences converted to compound word tokens comprising seven features (tempo, chord, bar-beat, position, pitch, duration, velocity) plus a type feature, embedded and projected via a spike-based encoder with LIF neurons.

Output: Generated symbolic music sequences represented as compound word tokens or spike trains corresponding to the input format.

Scoring recipe

def evaluate(predictions, gold=None):
    # Objective metrics (calculated per sequence)
    obj_scores = {
        'PC': count_unique_pitches(predictions),
        'IOI': calculate_avg_onset_intervals(predictions),
        'PCS': compute_consonance(predictions),
        # ... other objective metrics from Table A3
    }
    # Subjective metrics (collected via online platform)
    subj_scores = collect_human_ratings(predictions, groups=['Normal', 'Amateur', 'Expert'])
    # Cognitive metrics
    cognitive = {
        'impression': subj_scores['Q11'],
        'autobiographical_association': subj_scores['Q12'],
        'personal_preference': subj_scores['Q13']
    }
    return obj_scores, cognitive

Common pitfalls

• Objective statistical metrics often misalign with human perceptual judgments of musical quality, leading to over-optimistic model rankings.
• Cognitive-level metrics (impression, autobiographical association) are novel and may be overlooked or misunderstood by evaluators unfamiliar with the framework.
• Expert listeners demonstrate significantly higher tolerance for AI-generated music compared to normal/amateur listeners, which can skew subjective scores if listener demographics are not controlled.

Evidence (verbatim from paper)

The objective metrics in this paper were chosen based on previous studies to ensure a balanced and widely accepted assessment of model performance, which contains the pitch-related, rhythm-related and harmony-related categories: Pitch-related metrics includes pitch count (PC), Pitch Range (PR), Average Pitch Interval(PI), Pitch Entropy (PE), Pitch Class Entropy (PCE), Pitch-in-scale rate (PSR) and Polyphony (Pol); Rhythm-related metrics contains Average inter-onset interval (IOI), Note Length Transition Matrix (NLTM), Empty-Beat Rate (EBR), Groove Consistency (GC); Harmony-related metrics comprises Pitch consonance score (PCS) and Chord tone to non-chord tone ratio (CTnCTR).

Citation

@misc{liang2025muspike,
  title={MuSpike: A Benchmark and Evaluation Framework for Symbolic Music Generation with Spiking Neural Networks},
  author={Liang et al. (2025)},
  year={2025},
  note={arXiv:2508.19251}
}

• arXiv: 2508.19251