Sheet Music Benchmark Eval

sheet-music-benchmark-eval

Sheet Music Benchmark: Standardized Optical Music Recognition Evaluation — Martinez-Sevilla et al. (2025) (arXiv:2506.10488, 2025)

What this evaluates

Evaluates end-to-end optical music recognition (OMR) systems on their ability to transcribe scanned sheet music images into standardized **kern musical notation. It probes layout analysis, staff/page-level transcription accuracy, and robustness across diverse musical textures such as monophony, pianoform, and quartets.

Datasets

• Sheet Music Benchmark (SMB) — total 685; splits: test (685); HF PRAIG/SMB

Metrics

• OMR-NED (primary) — range: [0, 1]
  • Normalized Edit Distance between the predicted **kern sequence and the ground truth **kern sequence. It computes the minimum number of insertions, deletions, and substitutions required to transform the prediction into the gold standard, normalized by the length of the gold sequence, providing both a single performance score and granular error categorization by symbol type.

Input / output format

Input: Scanned sheet music images (full-page or staff/region-level bounding box crops).

Output: Text string in raw or standardized **kern format representing the transcribed musical notation.

Scoring recipe

def compute_omr_ned(pred_kern: str, gold_kern: str) -> float:
    # Compute standard Levenshtein edit distance
    edit_dist = levenshtein_distance(pred_kern, gold_kern)
    # Normalize by the length of the ground truth sequence
    n_ed = edit_dist / len(gold_kern)
    return n_ed

Common pitfalls

• Ambiguity in **kern encoding allows multiple valid representations for the same musical symbol, requiring strict tokenization (e.g., **ekern) or postprocessing to ensure fair comparison across models.
• Models may be evaluated at staff-level or full-page-level; mixing these granularities without accounting for layout analysis complexity skews results.
• The trade-off between vocabulary size and sequence length in standardized vs. extended **kern formats can artificially inflate or deflate edit distances if not handled consistently.

Evidence (verbatim from paper)

It is paired with the OMR Normalized Edit Distance (OMR-NED), a novel metric that provides both a single performance score and granular error analysis by categorizing failures at the level of notes, rests, beams, accidentals, and other notation elements, addressing a critical lack of standardized evaluation in OMR research.

Citation

@misc{martinezsevilla2025sheetmusicbenchmark,
  title={Sheet Music Benchmark: Standardized Optical Music Recognition Evaluation},
  author={Martinez-Sevilla et al. (2025)},
  year={2025},
  note={arXiv:2506.10488}
}

• arXiv: 2506.10488