Psrb Eval

psrb-eval

PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systems — Sedghiyeh et al. (2025) (arXiv:2505.21230, 2025)

What this evaluates

This benchmark evaluates the robustness and accuracy of automatic speech recognition (ASR) systems for the Persian language across diverse acoustic conditions, demographic groups, and linguistic domains. It specifically probes how well models handle regional accents, spontaneous or informal speech, and underrepresented demographics, while highlighting architectural and data-related performance gaps.

Datasets

• PSRB — total ?; splits: test (-1)

Metrics

• CER — range: percent
  • Character Error Rate; calculated as the minimum number of character insertions, deletions, and substitutions required to transform the predicted transcription into the ground truth, divided by the total number of characters in the reference.
• WER — range: percent
  • Word Error Rate; calculated as the minimum number of word insertions, deletions, and substitutions required to transform the predicted transcription into the ground truth, divided by the total number of words in the reference.
• SW-WER (primary) — range: percent
  • Substitution Weighted WER; a variant of WER that downweights minor and partial substitution errors to improve evaluation robustness, resulting in a lower score than standard WER when such errors are frequent.

Input / output format

Input: Raw Persian audio recordings spanning various domains, demographics, and acoustic conditions.

Output: Text transcription of the spoken Persian audio.

Scoring recipe

def evaluate_asr(predictions, references):
    cer_scores, wer_scores, sw_wer_scores = [], [], []
    for pred, ref in zip(predictions, references):
        cer_scores.append(cer(pred, ref))
        wer_scores.append(wer(pred, ref))
        sw_wer_scores.append(sw_wer(pred, ref))  # SW-WER downweights minor/partial substitutions
    return {
        'CER': sum(cer_scores) / len(cer_scores),
        'WER': sum(wer_scores) / len(wer_scores),
        'SW-WER': sum(sw_wer_scores) / len(sw_wer_scores)
    }

Common pitfalls

• Self-supervised models (e.g., Whisper, Seamless) are prone to hallucination errors on Persian, which standard WER may not fully penalize.
• SW-WER intentionally downweights minor and partial substitution errors compared to standard WER; directly comparing the two without accounting for this weighting leads to misinterpretation of model accuracy.
• Performance gaps between commercial and open-source models are driven primarily by data quality and quantity rather than architecture alone, so fine-tuning on diverse Persian data is critical.

Evidence (verbatim from paper)

For the evaluation of open-source models, Python scripts from each model’s repository were utilized to automatically generate results, which were then analyzed by measuring CER, WER, and SW-WER for each utterance and calculating their averages.

Citation

@misc{sedghiyeh2025psrb,
  title={PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systems},
  author={Sedghiyeh et al. (2025)},
  year={2025},
  note={arXiv:2505.21230}
}

• arXiv: 2505.21230