Spgispeech Eval

spgispeech-eval

SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition — O'Neill et al. (2021) (arXiv:2104.02014, 2021)

What this evaluates

Evaluates end-to-end speech-to-text models on financial domain audio, specifically testing their ability to produce fully formatted orthographic transcriptions including punctuation, capitalization, number denormalization, and disfluency handling. The benchmark measures how well acoustic architectures can learn text formatting directly from audio signals without relying on post-processing pipelines.

Datasets

• SPGISpeech — total 5000; splits: test (-1), val (-1); repo https://github.com/espnet/espnet/tree/master/egs2/spgispeech

Metrics

• WER (primary) — range: percent
  • Word Error Rate: (Substitutions + Deletions + Insertions) / Reference Words. Reported as a percentage on the test split.
• CER — range: percent
  • Character Error Rate: (Substitutions + Deletions + Insertions) / Reference Characters. Reported as a percentage on the test split.

Input / output format

Input: Raw audio recordings of financial speech

Output: Detokenized text string containing upper/lower case characters, digits, comma, period, apostrophe, hyphen, question mark, percent sign, and space

Scoring recipe

def compute_wer_cer(predictions, references):
    wers, cers = [], []
    for pred, ref in zip(predictions, references):
        s, d, i = levenshtein_ops(ref, pred)
        wers.append((s + d + i) / len(ref) if len(ref) > 0 else 0.0)
        cers.append((s + d + i) / len(ref) if len(ref) > 0 else 0.0)
    return sum(wers) / len(wers), sum(cers) / len(cers)

Common pitfalls

• Irreducible error exists due to subjective judgment in punctuation and disfluency handling in the ground truth.
• Normalization (lowercasing/filtering) significantly reduces error rates, so orthographic vs normalized results must not be compared directly.
• Models are evaluated using greedy decoding, not beam search, which may affect reported WER/CER.

Evidence (verbatim from paper)

As performance on the orthographic STT task hinges in large part on single-character distinctions such as capitalization and punctuation, we report both WER and CER for the test split of SPGISpeech, closely tracking the performance on the val split. We also estimate the respective error rates for a normalized transcription task by lowercasing the output and filtering all but letters, the apostrophe and the space character to conform to the conventional choice of STT vocabulary.

Citation

@misc{oneill2021spgispeech,
  title={SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition},
  author={O'Neill et al. (2021)},
  year={2021},
  note={arXiv:2104.02014}
}

• arXiv: 2104.02014