Jam Alt Eval

jam-alt-eval

Lyrics Transcription for Humans: A Readability-Aware Benchmark — Cífka et al. (2024) (arXiv:2408.06370, 2024)

What this evaluates

Evaluates automatic lyrics transcription (ALT) systems on readability-aware formatting, including punctuation, capitalization, line breaks, and background vocal annotations. It distinguishes errors by token type to measure how well models adhere to industry-standard musical semantics and prosodic structure.

Datasets

• Jam-ALT — total ?; splits: test (-1)
• Schubert Winterreise Dataset (SWD) — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: (substitutions + insertions + deletions) / reference length. Normalized by reference length.
• WER' — range: percent
  • Case-sensitive Word Error Rate: WER + case errors (hits with differing letter case counted as errors instead of hits).
• FP — range: percent
  • F-measure for punctuation tokens.
• FB — range: percent
  • F-measure for parentheses tokens.
• FL — range: percent
  • F-measure for line break tokens.
• FS — range: percent
  • F-measure for section break tokens.

Input / output format

Input: Raw audio track (optionally pre-processed with HTDemucs for vocal isolation) and song language identifier.

Output: Textual lyrics with enforced line breaks, capitalization, punctuation, and annotations for parentheses/section breaks, following readability-aware annotation guidelines.

Scoring recipe

def compute_metrics(pred, ref):
    # Tokenize into words and special formatting tokens (P, B, L, S)
    ops = edit_distance(pred, ref)
    wer = (ops.sub + ops.ins + ops.del) / len(ref)
    case_errors = count_case_mismatches(pred, ref)
    wer_prime = wer + case_errors / len(ref)
    fp = f1_score(pred.P, ref.P)
    fb = f1_score(pred.B, ref.B)
    fl = f1_score(pred.L, ref.L)
    fs = f1_score(pred.S, ref.S)
    return {'WER': wer, "WER'": wer_prime, 'FP': fp, 'FB': fb, 'FL': fl, 'FS': fs}

Common pitfalls

• Whisper models use stochastic decoding, so results must be averaged over multiple runs (e.g., 5) to be comparable.
• Providing the known language to Whisper often degrades performance compared to its auto-detection, contrary to typical ASR expectations.
• Using vocal separation (HTDemucs) as input can substantially degrade transcription accuracy for some models.
• The SWD subset contains obsolete German spelling that violates modern annotation guidelines, artificially inflating WER.

Evidence (verbatim from paper)

Table 1: Benchmark results (all metrics shown as percentages). WER is word error rate, $ ext{WER}'$ is case-sensitive WER, the rest are F-measures.

Citation

@misc{cifka2024lyricstranscription,
  title={Lyrics Transcription for Humans: A Readability-Aware Benchmark},
  author={Cífka et al. (2024)},
  year={2024},
  note={arXiv:2408.06370}
}

• arXiv: 2408.06370