Norwegian Asr Eval

norwegian-asr-eval

Boosting Norwegian Automatic Speech Recognition — de la Rosa et al. (2023) (arXiv:2307.01672, 2023)

What this evaluates

Evaluates automatic speech recognition (ASR) models on Norwegian Bokmål and Nynorsk transcriptions, measuring out-of-domain generalization and dialectal robustness across parliamentary and test speech corpora.

Datasets

• NPSC — total ?; splits: test (-1)
• NST — total ?; splits: test (-1)
• FLEURS (Norwegian) — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated as the sum of substitutions, deletions, and insertions divided by the total number of words in the reference transcription, reported as a percentage.

Input / output format

Input: Audio recordings of Norwegian speech paired with reference transcriptions.

Output: Predicted text transcription of the audio input.

Scoring recipe

def calculate_wer(predictions, references):
    # Align predicted and reference word sequences using dynamic programming
    # Count substitutions (S), deletions (D), insertions (I)
    N = len(references)  # total words in reference
    wer = (S + D + I) / N
    return wer * 100  # return as percentage

Common pitfalls

• WER scores in the paper are calculated on normalized text (numbers and times written out), whereas some baselines report WER on non-normalized text, making direct comparison misleading.
• Models are evaluated with and without a 5-gram language model, which significantly impacts WER; results must explicitly state whether an LM was used.
• Regional splits in the NST test set do not fully reflect Norway's dialectal diversity, potentially overestimating out-of-domain generalization.

Evidence (verbatim from paper)

We evaluate the performance of the models grouping their scores by the written language of the test sets in NPSC and NST. We report word error rates as percentages. For comparison purposes, we include the figures obtained in the original NPSC paper by Solberg and Ortiz (2022), as well as the work by Ortiz and Burud (2021) who also briefly evaluated ASR on NPSC.

Citation

@misc{delarosa2023boosting,
  title={Boosting Norwegian Automatic Speech Recognition},
  author={de la Rosa et al. (2023)},
  year={2023},
  note={arXiv:2307.01672}
}

• arXiv: 2307.01672