Tunisian Asr Eval

tunisian-asr-eval

LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect — Naouara et al. (2025) (arXiv:2504.02604, 2025)

What this evaluates

Evaluates automatic speech recognition (ASR) models on Tunisian Arabic dialect audio, measuring overall transcription accuracy and code-switching performance for embedded English and French phrases.

Datasets

• LinTO — total ?; splits: train (-1)
• TunSwitch — total ?; splits: test (-1)

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • Standard ASR metric: (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, N=number of reference words. Reported as a percentage.
• F1/Recall/Precision on Latin words — range: [0, 1]
  • Token-level F1, recall, and precision computed exclusively on Latin-script words (English/French) within code-switched references and predictions.

Input / output format

Input: Audio recordings of Tunisian Arabic speech, frequently code-switched with English or French.

Output: Transcribed text using an extended Buckwalter transliteration scheme that encodes both Arabic and Latin characters using ASCII pairs.

Scoring recipe

def compute_asr_metrics(preds, refs):
    # Standard WER
    wer = edit_distance(preds, refs) / len(refs.split())
    # Latin word metrics (token-level)
    pred_lat = [w for w in preds.split() if is_latin(w)]
    ref_lat = [w for w in refs.split() if is_latin(w)]
    tp = len([w for w in pred_lat if w in ref_lat])
    fp = len([w for w in pred_lat if w not in ref_lat])
    fn = len([w for w in ref_lat if w not in pred_lat])
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return {'wer': wer, 'latin_precision': prec, 'latin_recall': rec, 'latin_f1': f1}

Common pitfalls

• WER can exceed 100% due to high insertion rates, which the paper explicitly notes and should not be clipped or normalized.
• Latin word metrics are computed exclusively on code-switched portions, not the full transcript.
• Statistical confidence intervals are derived via bootstrap resampling, not analytical variance estimates.

Evidence (verbatim from paper)

Even if Whisper can recognize some Arabic, it fails to transcribe the Tunisian Arabic Dialect, with Word Error Rates (WER) ranging from 50% to more than 100% (indicating high insertion rates). To assess accuracy for recognition of English and French phrases in a code-switching context, we give F1, recall and precision scores on Latin words. All 95% confidence intervals are computed by performing bootstrap resampling.

Citation

@misc{naouara2025linto,
  title={LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect},
  author={Naouara et al. (2025)},
  year={2025},
  note={arXiv:2504.02604}
}

• arXiv: 2504.02604