qhjqhj00 hebrew-g2p-eval
Phonikud: Hebrew Grapheme-to-Phoneme Conversion for Real-Time Text-to-Speech — Yakov Kolani et al. (2025) (arXiv:2506.12311, 2025)
What this evaluates
This benchmark evaluates a model's ability to convert unvocalized Hebrew text into fully-specified IPA transcriptions, including accurate stress placement and shva realization. It also measures downstream text-to-speech quality and inference latency to assess real-time applicability.
Datasets
- ILSpeech — total ?; splits: test (-1)
- SASPEECH — total ?; splits: test (-1)
Metrics
WER(primary) — range: [0, 1]- Word Error Rate: (Substitutions + Deletions + Insertions) / Total Words. For TTS, computed by running ASR on generated audio and comparing to original text.
CER— range: [0, 1]- Character Error Rate: (Substitutions + Deletions + Insertions) / Total Characters. Computed similarly to WER.
WER^σ— range: [0, 1]- Word Error Rate disregarding stress mismatches. Stress diacritics are stripped from both predictions and gold before alignment.
RTF— range: ratio- Real-Time Factor: ratio of system processing/inference time to the duration of the generated audio. Lower is better.
Input / output format
Input: Unvocalized Hebrew text (consonantal script without diacritics).
Output: Fully-specified IPA transcription including vowel marks, stress diacritics, and shva realization.
Scoring recipe
def compute_wer_cer(pred, gold):
dist = levenshtein_distance(pred, gold)
return dist / len(gold) if len(gold) > 0 else 0.0
def compute_wer_sigma(pred_ipa, gold_ipa):
pred_clean = re.sub(r'[ˈˌ]', '', pred_ipa)
gold_clean = re.sub(r'[ˈˌ]', '', gold_ipa)
return compute_wer_cer(pred_clean, gold_clean)
def compute_rtf(inference_time_s, audio_duration_s):
return inference_time_s / audio_duration_s
Common pitfalls
- ASR-based WER/CER for TTS may not penalize phonetic inaccuracies (like stress) if the ASR model confidently transcribes the audio back to the original unvocalized text.
- RTF is calculated differently across baselines: open models use local CPU hardware, while proprietary models use cloud APIs, complicating direct latency comparisons.
- WER^σ explicitly ignores stress mismatches, which can mask critical phonetic errors that significantly impact TTS naturalness.
Evidence (verbatim from paper)
We calculate word-and character error rates (WER, CER) and WER when disregarding stress $(\mathrm{WER}^{\sigma})$ . We compare to two baselines: Firstly, we apply the existing SOTA Hebrew diacritizers DictaBERT (Shmidman et al., 2023) and Nakdimon (Gershuni and Pinter, 2022) with our IPA conversion, using reasonable defaults for ambiguous features (e.g. final stress, common in Hebrew).
Citation
@misc{kolani2025phonikud,
title={Phonikud: Hebrew Grapheme-to-Phoneme Conversion for Real-Time Text-to-Speech},
author={Yakov Kolani et al. (2025)},
year={2025},
note={arXiv:2506.12311}
}
- arXiv: 2506.12311