Ft Speech Asr Eval

ft-speech-asr-eval

FT Speech: Danish Parliament Speech Corpus — Kirkedal et al. (2020) (arXiv:2005.12368, 2020)

What this evaluates

Evaluates automatic speech recognition (ASR) systems on spontaneous, formal parliamentary speech in Danish. It tests both in-domain recognition accuracy and cross-domain transferability between the new FT Speech corpus and the established SBRead corpus.

Datasets

• FT Speech — total ?; splits: train (-1), dev (-1), test (-1)
• SBRead — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference transcript.

Input / output format

Input: 40-dimensional MFCC audio features (subsampled with iVector speaker adaptation) for acoustic modeling, combined with text transcripts for language modeling.

Output: Word-level transcribed text corresponding to the input audio sequence.

Scoring recipe

def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        # Compute edit distance (Levenshtein) between word lists
        errors = edit_distance(pred.split(), ref.split())
        total_errors += errors
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• Cross-domain evaluation mixes in-domain and out-of-domain test sets; failing to separate them obscures transferability results.
• Models trained on formal read speech (SBRead) perform poorly on spontaneous parliamentary speech, so domain mismatch must be explicitly reported rather than averaged.

Evidence (verbatim from paper)

ASR models trained on FT Speech achieve 14.01 WER and show strong transferability to Språkbanken data, while models trained on Språkbanken fail significantly on spontaneous speech, demonstrating FT Speech's value for realistic ASR development.

Citation

@misc{kirkedal2020ftspeech,
  title={FT Speech: Danish Parliament Speech Corpus},
  author={Kirkedal et al. (2020)},
  year={2020},
  note={arXiv:2005.12368}
}

• arXiv: 2005.12368