Loquacious Set Eval

loquacious-set-eval

Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Use — Parcollet et al. (2025) (arXiv:2505.21578, 2025)

What this evaluates

Evaluates automatic speech recognition (ASR) model performance across varying training data scales and model sizes, measuring generalization to in-domain and out-of-domain English speech benchmarks.

Datasets

• Loquacious Set — total ?; splits: val (-1), test (-1)
• Librispeech — total ?; splits: test-other (-1)
• Voxpopuli — total ?; splits: test (-1)
• CommonVoice — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference. Expressed as a percentage.

Input / output format

Input: Raw audio recordings paired with reference text transcripts.

Output: Decoded text transcript generated by the ASR model.

Scoring recipe

def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        # Standard Levenshtein distance alignment for word sequences
        edit_dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += edit_dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• The paper explicitly uses Librispeech test-other, not test-clean; confusing them leads to incorrect baseline comparisons.
• Decoding is performed without a language model; comparing these WERs directly to systems that use LMs will overstate the performance gap.
• WER is computed on raw word sequences without explicit normalization rules (e.g., case, punctuation, numbers) specified, which can cause variance across different evaluation scripts.

Evidence (verbatim from paper)

The obtained ASR performance are reported in Table [2]. First of all, and in line with the literature and overall consensus, it is fairly clear that both adding more data and increasing the size of the ASR model lead to reduced word error rates. For instance, on the test set of the Loquacious Set, the WER drops from 24.3% to 7.5% when switching from the small data split with 25M parameters conformer to the full dataset and 480M parameters.

Citation

@misc{parcollet2025loquacious,
  title={Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Use},
  author={Parcollet et al. (2025)},
  year={2025},
  note={arXiv:2505.21578}
}

• arXiv: 2505.21578