Spgispeech 2.0 Eval

spgispeech-2.0-eval

SPGISpeech 2.0: Transcribed multi-speaker financial audio for speaker-tagged transcription — Grossman et al. (2025) (arXiv:2508.05554, 2025)

What this evaluates

Evaluates end-to-end speaker-tagged automatic speech recognition (ASR) and speaker diarization on financial domain audio. It probes a model's ability to accurately transcribe speech while correctly assigning speaker identities to utterance segments in multi-speaker conversations.

Datasets

• SPGISpeech 2.0 — total 3780; splits: test (-1)

Metrics

• cpWER (primary) — range: percent
  • Concatenated Minimum Permutation Word Error Rate. Calculated by finding the optimal permutation of predicted speaker segments to reference transcripts across all possible permutations, reporting the minimum WER achieved.
• WER — range: percent
  • Word Error Rate. Calculated as substitutions, deletions, and insertions divided by the number of reference words.

Input / output format

Input: Multi-speaker financial audio recordings (50–90 second snippets) with orthographically normalized transcriptions and per-word timestamps.

Output: Transcribed text with speaker tags/identifiers assigned to each utterance segment, optionally with punctuation and capitalization.

Scoring recipe

def compute_wer(pred_words, ref_words):
    return (substitutions + deletions + insertions) / len(ref_words)

def compute_cpWER(pred_segments, ref_segments):
    min_wer = float('inf')
    for perm in permutations(ref_segments):
        wer = compute_wer(concat(pred_words), concat(perm.words))
        min_wer = min(min_wer, wer)
    return min_wer

Common pitfalls

• Confusing standard WER with cpWER; cpWER requires optimizing over speaker label permutations to account for diarization mismatches.
• Failing to report results under both With-PnC and Without-PnC settings, as performance varies significantly between them.
• Assuming speaker IDs are consistent across datasets; the permutation optimization handles arbitrary label mappings.

Evidence (verbatim from paper)

To show the accuracy on speaker-ID (speaker tagging), we use Concatenated Minimum Permutation Word Error Rate (cpWER) [25], which can reflect speaker-tagging accuracy in the form of Word Error Rate (WER). We define WER in a standard fashion as substitutions, deletions, and insertions divided by the number of reference words. cpWER is then calculated by finding the optimal permutation of predicted speaker segments to reference transcripts across all possible permutations, reporting the minimum WER achieved from the optimal permutation.

Citation

@misc{grossman2025spgispeech2,
  title={SPGISpeech 2.0: Transcribed multi-speaker financial audio for speaker-tagged transcription},
  author={Grossman et al. (2025)},
  year={2025},
  note={arXiv:2508.05554}
}

• arXiv: 2508.05554